
%
%  $Description: Author guidelines and sample document in LaTeX 2.09$ 
%
%  $Author: ienne $
%  $Date: 1995/09/15 15:20:59 $
%  $Revision: 1.4 $
%

\documentclass[times, 10pt,twocolumn]{article} 
\usepackage{latex8}
\usepackage{times,url}
\usepackage{alltt}
\newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
\newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\tion}[1]{\S\ref{tion:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{itemize}
   \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{itemize}}
\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}


%\documentstyle[times,art10,twocolumn,latex8]{article}

%------------------------------------------------------------------------- 
% take the % away on next line to produce the final camera-ready version 
%\pagestyle{empty}

%------------------------------------------------------------------------- 
\begin{document}

\title{How to (Sometimes) Avoid Drastic Process Change (using AI)
}
\author{Tim Menzies\thanks{
This research was conducted at
WVU and USC
under a NASA sub-contract.
Reference herein to
any specific commercial product, process, or service by trade name,
trademark, manufacturer, or otherwise, does not constitute or imply
its endorsement by the United States 
Government.}\thanks{Submitted to ICSE'09, \today }\thanks{Download at
\protect\url{http://menzies.us/pdf/08drastic.pdf}.},  Steve Williams,\\
Oussama El-Rawas\\
CSEE, WVU, USA\\
tim@menzies.us \\swilli12@mix.wvu.edu\\
oelrawas@mix.wvu.edu
\and
Barry Boehm\\
       Computer Science Dept.\\
 Uni. Southern California, USA\\
       boehm@sunset.usc.edu
\and
Jairus Hihn\\
       Jet Propulsion Laboratory\\
       California, USA\\
       jairus.hihn@jpl.nasa.gov
}


\maketitle
\thispagestyle{empty}

\begin{abstract}
Predictors for software development time,  effort, and quality
usually use local data to tune their predictions. 
Such 
local tuning data is often unavailable. 
Without a tuned process model, it is had to rank process options 
with a project.
This inhibits effective debates
about
process change.
For some process models,
the performance
variance due to tuning uncertainty can be 
greatly reduced by AI search methods
selecting
process options.
That is, even without local tuning,
AI search can comparatively assess (a)~proposed
drastic process change against (b)~other possibilities.
To demonstrate this, we compare drastic process
changes against
far less disruptive changes to a project. In all most cases,
our AI search engine found combinations of current project options
that out-performed those drastic cases.
\end{abstract}



%------------------------------------------------------------------------- 
\Section{Introduction}

Drastic process change is highly disruptive. 
For example,
your manager has returned from ICSE'09
excited about the next new thing in software engineering.
``We need to drastically alter our projects,'' she says, ``in order to support (say) automatic
formal methods, agile process,  etc.''. 
This change
may entail switching from Windows to LINUX (since there are more 
research prototypes for automated formal methods running on LINUX);
stopping production
where engineers retrain;  or, worse, firing the engineers and making new hires. 
Alternatively,
your manager knows that a budget cut is imminent and 
wants to discuss drastic reduction in project scope.
This change may
alter projected revenue streams 
(since
the next release will have less features and hence less market
appeal). Changing revenue projections can be very detrimental to
the rate
of investor capital received by a software development company.

\begin{figure}[!b]
\footnotesize
\begin{tabular}{|c@{~}p{.4\linewidth}|p{.5\linewidth}|}\hline
&Drastic change & Possible undesirable impact\\\hline
1&Improve personnel & Firing and re-hiring personnel.\\\hline
2&Improve tools, techniques, or  
development platform & Changing operating systems.\\\hline
3&Improve precedentness / development flexibility &
Changing the goals
of the project and the development method.\\\hline
4&Increase architectural analysis /  risk resolution &
Far more elaborate early life cycle analysis.\\\hline
5&Relax schedule  & Delivering the system later\\\hline
6&Improve process maturity & May be expensive in the short term.\\\hline
7&Reduce 
functionality & Delivering less than expected.\\\hline
8&Improve the team & Requires effort on team building\\\hline

9&Reduce quality 
& Less user approval, smaller market.\\\hline
\end{tabular}
\caption{Nine drastic changes from~\cite{boehm99}.}\label{fig:dchange}
\end{figure}
\newcommand{\boxplot}[5]{
\begin{picture}(100, 7)
\put(#1, 4){\line(1, 0){#2}}
\put(#3, 4){\circle*{4}}
\put(#4, 4){\line(1, 0){#5}}
\put(50, 0){\line(0, 1){7}}
\end{picture}
}
\fig{dchange} lists undesirable side-effects of nine kinds of drastic
changes.
The management challenge is know
when the possible drawbacks of drastic change  are justified.
For this task,
software process models
can be used to run multiple ``what-if'' scenarios that return the least changes
to a project  that most improves software quality, most decreases development time, and
most decreases development effort.

Process ``what-if'' analysis  
is hindered by the {\em local tuning problem}.
Software process predictions are 
most accurate {\em after} models have been
tuned to local data. Unfortunately, 
   local tuning data is difficult to obtain~\cite{me06d}.  
This is due to  data not being collected or
the business sensitivity associated
   with the data as well as differences in how the metrics are defined,
   collected and archived.  

The premise of our research is that,
   in terms of ranking process options,
precise local tunings may  not be necessary.
Rather, knowledge of local circumstances may suffice.
Based on research dating back to 1981~\cite{boehm81}, we assert that the
space of {\em
possible tunings} is well known. This space can be explored 
using AI search within the current project structure
to
minimizes effort, time, defects, and threats. 

Such an AI search automatically explores and ranks a very large space of options.
Note that the results of such an AI search is {\em 
not} a precise estimate. Rather,
given a set of possible process changes, we can {\em
sort and evaluate the relative merits}
of different sets of changes. 
That is, our AI search engine is a {\em planner}, and not necessarily a {\em predictor}.

%We contrast the results of {\em drastic change} and  {\em AI search} with the current project structure
%as follows.
%After an AI search within the bounds of the current project,
%the resulting recommendations can be implemented by the manager running that project.
%For example, the client may be happy with reliability levels from moderate to high
%and the project manager is free to adjust the intended reliability to moderate.
%On the other hand, {\em drastic change} 
%requires the involvement of senior management since it
%falls outside of the current definition of a project.
%For example, a drastic change might require firing the entire current development
%team.
%
This paper comparatively ranks the value of drastic change
or 
  AI search within
current project bounds. For four NASA case studies,
our
main result is that:
\begin{quote}
{\em In  a majority of experiments, AI search  within the current structure of a project
does as well or better than
drastic change}.
\end{quote}

%XXX related work. bayes etc.

For example, \fig{results}
shows the range of
defect/KLOC  predictions seen in 100 Monte Carlo
simulations (where exact values are selected at random
from the space of possible values). 
The distributions are different due to different project constraints
applied. The constraints learned by AI search within current project bounds resulted
in much smaller defect predictions that any other approach (see line~1). 
That is,
all the defect reduction arising from   drastic change can also be achieved
by simpler AI search.

%More formally, we say that we if a model exhibits three properties, then
%we can control it without local tuning data.
%A model converts inputs to outputs using tuned internal parameters; i.e.
%\[Out=model(In,Tunings)\]
%The output mean
%   $\mu$ and variance $\sigma$ comes from
%the mean and variance of $In$   ($\mu_{I},\sigma_{I}$) and
%the space of possible $Tunings$ ($\mu_T,\sigma_T$).
%Model tuning takes historical data to change $\mu_T$ and reduce $\sigma_T$
%and, clearly, untuned models have large $\sigma_T$ and hence large output $\sigma$.
%If the goal is to reduce model output (property1)
%and if model
%output variance is reduced by the mean
%(i.e.  property2: $\mu \propto \sigma$)
%then (a)~minimizing 
%mean output values has the desirable side-effect of also reducing the variance; and
%(b)~the variance added by 
%sampling across $\{\mu_T,\sigma_T\}$ can be selecting 
%appropriate input values $selected \subseteq I$. 
%Also, 
%for models where the effects on the output of $I$ are greater than $T$ (property3), then 
%the output $\mu$ can be minimized by using the right input selections.
%
%The
%empirical results presented below (and in~\cite{me08c,me07f}) are consistent with the hypothesis
%that commonly used software process models (specifically, the USC COCOMO suite of models)
%have these three properties; i.e. they can be controlled by AI search within the space of current project options.
\begin{figure}[!b]
\footnotesize
\begin{tabular}{rl}
Change    & defects/KLOC (median predictions, 100 runs)\\\hline
AI search &  \rule{3mm}{1mm} 6.7\\
 Improve pcap  & \rule{11mm}{1mm} 21     \\
 Improve tool/tech/plat  & \rule{15mm}{1mm} 31\\
 Reduce functionality  & \rule{18mm}{1mm} 36\\
 Improve prec/flex  & \rule{27mm}{1mm} 55\\
 Arch/risk resolution & \rule{28mm}{1mm} 56\\
 Relax schedule  & \rule{32mm}{1mm} 64\\
 Improve pmat & \rule{34mm}{1mm} 67\\
 Improve team  & \rule{34mm}{1mm} 68\\
Do nothing& \rule{42mm}{1mm} 85\\
Reduce quality  & \rule{47mm}{1mm} 94
\end{tabular}
\caption{
Median defect predictions after making changes to 
to a ``Baseline''.
As might be expected, ``Reducing quality'' (last line) results in more defects than
doing nothing.
}\label{fig:results}
\end{figure}
\input{cocomo}


This result should be welcome news to any software manager struggling to maintain
control and continuity over their project in the face of multiple demands
for drastic change.
Note that we {\em do not} claim that
AI search {\em always} beats drastic change.
We offer  examples below where
extremely drastic change beats AI search.
Nevertheless,  at the very least, our results suggest that project managers
may have more project improvement options than they currently realize.
Hence, before performing drastic
changes, it can be worthwhile to take use AI search to explore
the available 
options within the current structure of  a project.

The rest of this paper is structured as follows.
After notes on prior work, we discuss with examples the options
available within the current structure of four sample projects.
The
drastic changes of \fig{dchange} will then be implemented as 
operators on those projects.
Next, we discuss various models used in this study and review the space
of possible tunings within those models. 
This is followed by an overview of the
SEESAW AI search engine used to explore the input space of the models. 
Finally, empirical results will be presented comparing the results
of AI search with SEESAW versus applying drastic changes of \fig{dchange}.

\Section{Background }

For several decades, one of us (Boehm) has worked on a variety of software process models including:
\bi
\item
The  COCOMO effort and time predictor~\cite[p29-57]{boehm00b}; 
\item
The COQUALMO defect~\cite[p254-268]{boehm00b};
\item
The THREAT  predictor  project overruns~\cite[284-291]{boehm00b}.
\ei
One advantage of these models is that,
unlike other models such as PRICE-S~\cite{park88}, 
SLIM~\cite{putnam92}, or SEER-SEM~\cite{jensen83}
our models are
fully described in the literature. Also, at least
for the effort model, there exists  baseline results~\cite{chulani99}. 
\fig{cparems} lists the features of those  models.

Most of the COCOMO literature
concerns 
 {\em manual use} of these models for decision making or
{\em automatic model tuning}.
An alternative approach is the use of
AI search and the above models to explore {\em automatic} decision making
{\em without local tuning}.
At ASE'07~\cite{me07f}
we used a simulated annealing search engine~\cite{kirkpatrick83}
to explore a very focused question: when is it
useful to use automated software engineering tools? 
While that question is of great interest to the automated software engineering
community, here we explore a question 
that is relevant to a much broader audience: the relative merits
of AI search versus drastic project changes.

The reader may wonder why we use an AI search. Would not a simpler method suffice?
As discussed below, many of the relationships inside the COCOMO/COQUALMO
models are linear.
For such models, AI search is needlessly complex and 
a simple
linear extrapolation
will suffice to extend  
from some current situation into some possible future.

Linear extrapolation
assumes that the gradient of
the linear dimensions is known with certainty. This is not necessarily the case, especially with our models.
For example, the COCOMO effort model predictions are effected
linearly and exponentially by two features $a,b$.
Baker~\cite{baker07}
studied the range of $a,b$ learned from thirty 90\% random samples of some NASA data
The ranges varied widely:
\begin{equation}\label{eq:ab}
\left( 2.2 \le a \le 9.18 \right) \wedge \left(0.88 \le b \le 1.09\right) 
\end{equation} 
That is, any simple linear extrapolation over COCOMO models could be highly misleading.
Elsewhere we have tried taming the variance of \eq{ab}
with feature subset selection (FSS)~\cite{chen05,me06d} or more
data collection. Each approach was only partially
successful. 
FSS reduces, but not elminates the $\{a,b\}$ variance. Also, further
data collection is possible, but only at great organizational expense. 
For example, after two years we were only able to
add 7 records to a NASA-wide software cost metrics repository~\cite{me07f}.

Another issue with simple linear extrapolation is  that,
when optimizing for effort {\em and} time
{\em and} defects,
there may be contradictory effects.
For example, the results shown below illustrate
a situation where effort was {\em reduced}, but defects {\em increased dramatically}. Hence, optimizing
our models is not a simple matter of moving fixed distances over some linear effect: there must also be
some trade-offs considered.

These two issues lead to the design of SEESAW.
Historically, it was the discovery of these large $a,b$ variances that moved this research team to explore
what stable conclusions can be found over wide ranges of possible tunings. Also, AI search methods like
SEESAW  are very useful for exploring competing constraints such as effort vs schedule.

In his text {\em Empirical Methods in 
Artificial Intelligence}, Cohen~\cite{cohen95}
advises baselining a supposedly more sophisticated
method against simpler alternatives. 
A simpler alternative to automatic AI search is 
the manual application of known best
practices in improving  a project. 
Before we endorse SEESAW or any other search algorithm,
it is appropriate to first test
if these engines do better than accepted best practices.

Previously~\cite{boehm99}, we have reviewed  nine 
such practices that can be manually applied- see the list   in \fig{dchange}.
This paper performs Cohen's baseline test 
and compares the effects of applying accepted best practices versus the
changes proposed by the SEESAW AI search engine.

\Section{Options Within Projects}
Our thesis is that AI search with SEESAW 
find options within the current structure of
a project that can control even untuned models. Model tuning and
AI search are discussed below. Before exploring that issue, we first explore
options within the current structure of a project.

\begin{figure}[!b]
\renewcommand{\baselinestretch}{0.95}
\begin{center}
\scriptsize
\begin{tabular}{c|lrr|lr}
       &\multicolumn{3}{c|}{ranges}      &\multicolumn{2}{c}{values}\\\hline
project&feature&low&high&feature&setting\\\hline
&prec&1&2&data&3\\
OSP:&flex&2&5&pvol&2\\
Orbital&resl&1&3&rely&5\\
space&team&2&3&pcap&3\\
plane&pmat&1&4&plex&3\\
&stor&3&5&site&3\\
&ruse&2&4&&\\
&docu&2&4&&\\
&acap&2&3&&\\
&pcon&2&3&&\\
&apex&2&3&&\\
&ltex&2&4&&\\
&tool&2&3&&\\
&sced&1&3&&\\
&cplx&5&6\\
&KSLOC&75&125\\
\multicolumn{6}{c}{~}\\\hline
&prec&3&5&flex&3\\
OSP2 &pmat&4&5&resl&4\\
&docu&3&4&team&3\\
&ltex&2&5&time&3\\
&sced&2&4&stor&3\\
&KSLOC&75&125&data&4\\
&&&&pvol&3\\
&&&&ruse&4\\
&&&&rely&5\\
&&&&acap&4\\
&&&&pcap&3\\
&&&&pcon&3\\
&&&&apex&4\\
&&&&plex&4\\
&&&&tool&5\\
&&&&cplx&4\\
&&&&site&6\\
\multicolumn{6}{c}{~}\\\hline
&rely&3&5&tool&2\\
JPL &data&2&3&sced&3\\
flight&cplx&3&6&&\\
software&time&3&4&&\\
&stor&3&4&&\\
&acap&3&5&&\\
&apex&2&5&&\\
&pcap&3&5&&\\
&plex&1&4&&\\
&ltex&1&4&&\\
&pmat&2&3&&\\
&KSLOC&7&418&&\\
\multicolumn{6}{c}{~}\\\hline
&rely&1&4&tool&2\\
JPL &data&2&3&sced&3\\
ground&cplx&1&4&&\\
software&time&3&4&&\\
&stor&3&4&&\\
&acap&3&5&&\\
&apex&2&5&&\\
&pcap&3&5&&\\
&plex&1&4&&\\
&ltex&1&4&&\\
&pmat&2&3&&\\
&KSLOC&11&392&&\\
\end{tabular}
\end{center}
\caption{Four case studies.
The numeric values here map to COCOMO inputs;
i.e. $\{1,2,3,4,5,6\}$ maps to $\{very low, low, nominal,
high, very high, extra high\}$ (respectively).}\label{fig:cases}
\end{figure}
Software process control is complicated by uncertainties within a project.
Exactly how good are our analysts? How large will be the final system? Do we really
understand our tools? These are just some of the questions that most project managers
cannot answer precisely.
For example, 
\fig{cparems} defines the features used in this study and \fig{cases} described
four NASA projects using those features:
\bi
\item OSP is the GNC (guidance, navigation, and control)
component of NASA's 1990s {\em Orbital Space Plane} experiment;
\item OSP2 is a later version of OSP;
\item Flight and ground systems reflect typical ranges seen at NASA's Jet Propulsion Laboratory. 
\ei
Some of the features in \fig{cases}
are known precisely; i.e. see all the features with single {\em values}.
But  many of the features in \fig{cases} do not
have precise values;
i.e. see all the features that {\em range} from some {\em low} to {\em high}
value.  Sometimes, the ranges
are very narrow (e.g. the process maturity of JPL ground software is between 2
and 3) and  sometimes, the ranges are very broad.
\fig{cases} does not mention all the  features listed in \fig{cparems}
inputs.  For example, our defect predictor has inputs for
use of {\em automated analysis}, {\em peer
reviews}, and {\em execution-based testing tools}. For all inputs
not mentioned in \fig{cases}, values are picked at random from very low
to very high.

\Section{Imposing Drastic Changes}
Using \fig{cases}, we can now offer a more precise definition of 
{\em drastic change} and {\em AI search} within the boundaries
of a current project.
We say that a drastic change is a reorganization to a project {\em outside} of the ranges
shown in \fig{cases} while {\em AI search} is reorganizing a project {\em within}
those ranges.
AI search just moves the project into some subset of the current project assumptions
while drastic changes alters the current assumptions of a project.
For example, one drastic change from \fig{dchange} is to increase process maturity.
According to \fig{cases}, JPL ground systems have process maturities in the range 2 to 3
(low to nominal). Drastic change \#6 from \fig{dchange}
is ``improve process maturity''. 
Demanding very high
CMM levels for these JPL ground systems would be a drastic change.
Such a change would take years,
would require the commitment  of very senior JPL management, and may necessitate
extensive
retraining local staff and/or hiring of new staff.

\begin{figure}[!t]
\footnotesize
\begin{tabular}{|c@{~}p{.5\linewidth}|p{.4\linewidth}|}\hline
&Drastic change 
& Changes to \fig{cases} \\\hline

1&Improve personnel 
& 
acap = 5; pcap = 5;
                    pcon = 5 \newline 
                    apex  = 5 ;
                    plex = 5 ;
                    ltex = 5 \\\hline
2&Improve tools, techniques, or  development platform 
& 
    time = 3; 
                    stor = 3\newline
                    pvol = 2;
                    tool = 5\newline
                    site = 6\\\hline
3&Improve precedentness / development flexibility 
&
prec = 5;
flex = 5
\\\hline
4&Increase architectural analysis /  risk resolution 
&resl = 5
\\\hline
5&Relax schedule  
& 
    sced = 5      
\\\hline

6&Improve process maturity 
& pmat = 5
\\\hline

7&Reduce \newline
functionality 
& 
 data = 2; 
                    kloc  * 0.5
\\\hline
8&Improve the team 
&
team = 5 
\\\hline

9&Reduce quality 
& 
  rely = 1 ;
                    docu = 1\newline
                    time = 3 ;
                    cplx = 1
\\\hline

\end{tabular}
\caption{Implementing drastic changes.
}\label{fig:how}
\end{figure}

\fig{how} defines the values we imposed on each case study as part
of each drastic change. 
Most of the values in \fig{how} are self-explanatory with two exceptions.
Firstly, the 
$kloc*0.5$ in ``reduce functionality'' means that  when imposing this drastic change,
we only implement half the system.
Secondly, most of the features fall in the range one to five.
However, some have minimum values of 2 (e.g. $pvol$ in ``improve tools/tech/dev'')
and some have maximum values of 6 (e.g. $site$ in ``improve tools/tech/dev'').
This is why some of the drastic changes do not always run to one or five.

These changes were defined by reflecting over our prior publication~\cite{boehm99}
and mapping that paper into the same syntax as the  inputs of \fig{cases}.
To impose a drastic change on a case study,
then  if that change refers to feature $X$ (in the right hand side column of
\fig{how}),
then we
first
(a)~removed  from the values and ranges of the case study (if it is present); then
(b)~added the changes of \fig{how} as fixed value(s) for that case study.

\Section{Finding Alternatives to Drastic Change}
The premise of this paper is that drastic change is disruptive. If possible,
such
drastic change 
should be avoided in favor of AI search; i.e. changes taken only from the current
project structure. 

The current project structure of our four case studies is defined in the ranges
of  \fig{how}.
We use an AI search engine to find subsets of those ranges
that most
reduce defects, development effort, development time, and threats.

\input{seesaw}

\input{model}
\input{results1.tex}
\Section{Results}
Figures \ref{fig:effort}, \ref{fig:time}, \ref{fig:defect} compares the effort, time,
and defect predictions
obtained after either:
\bi
\item
Applying
the drastic changes of \fig{dchange};
\item Or applying the recommendations found by
AI search within the current project ranges;
\item Or doing nothing at all and just running a Monte Carlo simulation
across the ranges of \fig{cases}.
\ei
For reasons of space, the threat results are not shown. The important result trends
discussed below also hold for the threat results. Hence, there is only minimal added value
in considering those results.

In the result figures:
\bi
\item
The rows are sorted by the median performance score.
\item
The ranks shown in the left-hand-column are computed from a Mann-Whitney 95\% confidence 
test\footnote{This test was chosen since
(a) due to the random nature of
Monte Carlo simulations, the inputs to each run are not paired;
(b) ranked tests make no, possibly inappropriate, assumption about normality of the results.};
two rows have the same rank 
if there is 
no statistical difference in their distributions. 
\item
The ``Do nothing '' row  comes from  Monte Carlo simulations of the current ranges, without any changes.
\ei
Each row in the results is a {\em horizontal quartile charts} representing the 
results of 100 
{\tt run} (see \fig{MWF}): 
\bi
\item Each row shows the full range, minimum to maximum, of the scores collected during the simulation.
\item The median result is shown as a large black dot. 
\item
These dots are
surrounded  left and right
by
black lines showing bottom/top quartiles, respectively (so the two middle quartiles
are represented as white lines left and right of the black dot).
\item
The left-hand-side of each row denotes $score=0$.
All the performance scores (effort, defects, time) get {\em better} when
the observed scores get {\em smaller}. 
That is, {\em better changes} are closest to zero.
\ei
Figures \ref{fig:effort}, \ref{fig:time}, \ref{fig:defect} show results from 12 experiments:
\[ 
\{effort, time, defects\}*\{flight, ground, OSP, OSP2\}\]
In $\frac{12}{12}$ (i.e. all) experiments, the median score from
AI search is better than half the other changes.
Better yet, in $\frac{6}{12}$ (i.e. half) experiments, AI search offers best predictions
while in two more (the defect results for OSP and OPS2), AI search ties in the top rank.
This result is the justification 
for our claim in the introduction that 
in  a majority of experiments $\frac{8}{12}$, AI search  within the current structure of a project
does as well or better than
drastic change.

The worst results from AI search come from OSP2:
\bi
\item
\fig{effort} shows the median effort for AI search to be near the
middle of all changes. 
\item \fig{defect} shows that the largest variance
for any change measured on any scoring dimension is found by AI
search (observe the very wide ranging OSP2 results for ``AI search'').  
\ei
This result is hardly surprising.
 Recall from \fig{cases} that
OSP2 is the most constrained case study explored by this
paper. For the most part,
the features all have fixed values. SEESAW works by exploring subsets of the available options.
When that option space is limited, the benefits of SEESAW are also limited:
The lesson of OSP2 is  that we cannot expect an AI search engine to find 
significant options unless we give first given them enough  options to explore.

In other cases where AI search does worse than other changes, some of those changes
are so dramatic that most managers would reject them. For example,
in the effort and time results, AI search is beaten by ``reduce
quality'' in $\frac{4}{12}$ experiments.  The benefits of these
effort and time reductions should be weighed carefully against their
implications on defects. \fig{defect} shows that ``reduce quality''
results in the largest number of defects seen in any change. While
there may exist projects that are willing to deliver code with large
numbers of defects, we believe that usually this is unacceptable.

Another drastic change that defeats AI search is ``improve pcap'' (personnel capabilities).
Recalling \fig{how}, we see that improving personnel capability requires changing
two analyst features, three personnel features, as well as addressing some issues relating
to language and tool experience. Of all the drastic changes studied in this paper, this
change is the most complex. It might also be the hardest to implement since it requires
dramatic change to the current personnel working on the project (perhaps even firing old
staff and hiring new ones).
If an organization can handle such large scale drastic change, then in some
cases (e..g. 
in $\frac{4}{12}$ experiments from 
Figures \ref{fig:effort}, \ref{fig:time}, \ref{fig:defect})
then that might indeed
out-perform SEESAW.
However, most managers might want to avoid such complex organization change, preferring
instead fine tuning the structure of their current project using AI search.

\Section{Conclusion}
In summary, AI search does not always defect draft change. However, it works often enough
for us to recommend to managers that before they 
resort to drastic measures, they should first check if AI search can find
some tactical restructuring within the bounds of the current project.

XXX assume we can change stuff

Search engines.

Software project managers may have more effective change options than they realize and AI search
can find those options. Fast. Under 5 seconds.

AI search does well (always in the top half), in $|frac{6}{12}$ cases, top method. Sometimes fails, OSP2.
if you fix everything there is nothing left to fix.

\bibliographystyle{latex8}
\bibliography{refs}

\end{document}
\underline{{\bf KEY:}}\\
The quartile charts, shown on left,
plot the defects predictions after either
AI search with  AI search (line one);
or  applying various drastic changes; or 
after 
doing nothing (second last line).
\bi
\item
Results are sorted on 
median results from 100 Monte Carlo simulations.
\item
The ``Baseline'' row (second from the bottom) shows the defects predicted from the projects,
without any additional constraints.
\item
Median results are  shown as black circles. These
medians are surrounded  left and right
by
black lines showing bottom/top quartiles (respectively).
\item
The ranks shown in the left-hand-column are computed from a Mann-Whitney 95\% confidence test:
two rows have the same rank 
if there is 
no statistical difference in their distributions. 
For example, the
four rows have the same rank since their distributions are essentially the same.
\ei\\\hline
\end{tabular}
\end{minipage}
\caption{AI search (see line 1) reduces defects
far more than 
drastic process changes.
As might be expected, ``reducing quality'' 
produces more defects than
``Baseline'' (see last last).

