In this chapter, we will present the anti-automation bias present in the software engineering community. This is followed by a study that attempts to argue against this bias by presenting a business case for using automated software engineering (ASE) tools. Note that this study was initially conducted for and reported in {\bf ASE'07}:  {\em The business case for automated software engineering}. \cite{me07f}

\section{Sociology vs. Tools}
Much current ASE research concerns
{\em automatic analysis} of source code or better
{\em execution-based testing tools},
These tools might, say,
verify formal properties or
search for the fewest regression tests that exercise  most of the system.
Some of these
tools are ready for industrial use; e.g, SPIN~\cite{holzmann97} or 
JPF~\cite{have00}, just to name
a few.   

We seeked to make the business case for introducing some
of these new ASE tools into large NASA projects. This case proved difficult to make,
due to an {\em anti-automation bias} and the {\em local tuning problem}.

\begin{figure}[!b]
\scriptsize
\begin{center}
\begin{tabular}{l@{}|l@{~}r@{~}l}
%& & relative\\
id & features  & relative weight\\\hline
1&Personnel/team capability  & 3.53	&\rule{17mm}{2mm}\\
2&  Product complexity & 2.38&	\rule{12mm}{2mm}\\
3& Time constraint & 1.63&	\rule{8mm}{2mm}\\
4&    Required software reliability & 1.54&	\rule{8mm}{2mm}\\
5&    Multi-site development   &	1.53&	\rule{7mm}{2mm}\\
6&       Doc. match to life cycle  & 	1.52	&\rule{7mm}{2mm}\\
7&  Personnel continuity  &	1.51&	\rule{7mm}{2mm}\\
8&  Applications experience & 1.51	&\rule{7mm}{2mm}\\
9& Use of software tools & 1.50	&\rule{7mm}{2mm}\\
10& Platform volatility & 1.49&	\rule{7mm}{2mm}\\
11& Storage constraint & 1.46&	\rule{7mm}{2mm}\\
12& Process maturity  &	1.43	&\rule{7mm}{2mm}\\
13&    Language \& tools experience  &	1.43	&\rule{7mm}{2mm}\\
14&     Required dev. schedule & 1.43	&\rule{7mm}{2mm}\\
15&Data base size & 1.42&	\rule{7mm}{2mm}\\
16&  Platform experience & 1.40&	\rule{7mm}{2mm}\\
17&     Arch. \& risk resolution  &	1.39&	\rule{7mm}{2mm}\\
18&  Precedentedness  &	1.33&	\rule{6mm}{2mm}\\
19&  Developed for reuse  	&1.31	&\rule{6mm}{2mm}\\
20& Team cohesion  &	1.29&	\rule{6mm}{2mm}\\
21&    Development mode   &	1.32	&\rule{6mm}{2mm}\\
22& Development flexibility &	1.26	&\rule{6mm}{2mm}\\
\end{tabular}
\end{center}
\caption[Relative effects of COCOMO attributes on development effort]{Relative effects of COCOMO attributes on development effort.
Data from
a regression analysis of 161 projects~\protect\cite{boehm00a}.  
}\label{fig:coceffects}
\end{figure}

The {\em anti-automation bias} was seen
at an ICSE 2007 panel.
Tim Lister (a co-author of Peopleware~\cite{demarco87})
commented
that ``sociology beats technology
in terms of successfully completing projects''- a notion endorsed by the other panelists.
That is,
software project managers should focus {\em less}
on new ASE tools
and {\em
more} on managing the sociology aspects of their team (e.g. decrease
staff turnover).

\fig{coceffects} offers some support for this bias.
This figure shows the known relative productivity effects
of changing project features.
According to this figure,
the benefits of automatic tools (ranked number nine in the list) can be equaled or bettered
via other means (e.g. any item 1 to 8 {\em or}  any {\em pair} of items 
10 to 22).

Note that this support for the anti-automation bias is based solely
on the development effort; i.e.
\fig{coceffects}
is blind to the impact of new ASE tools in reducing {\em defects}
and any  other {\em threats} to the success of the project.
A complete business case should therefore 
study
predictors for effort {\em and} defects {\em and} threats {\em and} schedule.

\section{NASA Experiments and ASE Tools}\label{sec:aseexp}

Before we can
assess the relative merits of new ASE tools 
versus other methods, we must first define ``new ASE tools'' and ``other methods''.
Such a definition can be achieved via the ontology of COCOMO, COQUALMO, SCHEDULE, and THREAT, shown in \fig{cparems}, and discussed in \tion{models}.
This figure lists a variety of project {\em features} with the 
{\em range}
\{very low, low, nominal, high, very high, extremely high\} or
\[\{vl=1, l=2, n=3, h=4, vh=5, xh=6\}\]
Lister's sociological features occur many times
within \fig{cparems}. For example, {\em team} refers to the sociology
within a development team and its cohesiveness while {\em pcon}
refers to the staff turnover rate within a project.
 Also listed in \fig{cparems}
are factors like {\em acap, pcap} representing analyst and programmer
capabilities (respectively).

As to technological features, new ASE tools
appear as
{\em execution-based testing tools (ett)} and
{\em automated analysis (aa)}.
Chulani~\cite{chulani99a} defines
the top half of 
{\em automated analysis} as:
\bi
\item[\textbf{4}] (\textbf{high}):
intermediate-level module and 
inter-module code syntax and 
semantic analysis. 
Simple requirements/design 
view consistency  checking. 
\item[\textbf{5}] (\textbf{very high}):
More elaborate 
requirements/design view 
consistency checking. 
Basic distributed-processing 
and temporal analysis, model 
checking, symbolic execution. 
\item[\textbf{6}] (\textbf{extremely high}):
Formalized\footnote{
Consistency-checkable pre- 
conditions and post-conditions, 
but not necessarily mathematical theorems.} 
specification and 
verification. 
Advanced distributed 
processing and temporal 
analysis, model checking, 
symbolic execution. 
\ei

\noindent
The upper half of {\em execution-based testing and tools} is:
\bi
\item[\textbf{4}] (\textbf{high}):
Well-defined test sequence 
tailored to organization 
(acceptance / alpha / beta / 
flight / etc.) test. 
Basic test coverage tools, test 
support system. 
\item[\textbf{5}] (\textbf{very high}):
More advanced test tools, test 
data preparation, basic test 
oracle support, distributed 
monitoring and analysis, 
assertion checking. 
Metrics-based test process 
management. 
\item[\textbf{6}] (\textbf{extremely high}):
Highly advanced tools for test 
oracles, distributed monitoring 
and analysis, assertion 
checking 
Integration of automated 
analysis and test tools. 
Model-based test process 
management. 
\ei

A  review of recent proceedings of the IEEE ASE conferences suggests
that 
a range of five or six in the above features includes
the kind of
``new ASE tools'' we mean to explore in this section.
Hence, to compare ``new ASE tools'' to ``other methods'',
we will try to reduce effort and defects and development time and threats
using  just
\[
\begin{array}{c}
\mbox{{\em automated analysis}}\in \{5,6\}\;\vee \\
\mbox{{\em execution-based testing and tools}} \in \{5,6\}
\end{array}
\]
or ``other methods'' (i.e. other ranges of \fig{cparems}).
\input{figures/nasa/allALLresults}
\input{figures/nasa/strategicALLresults}
\input{figures/nasa/tacticalALLresults}
 We ran STAR over the projects specified in \fig{cases} using the default, strategic and tactical (defined in \fig{strattact} and \tion{nasaprojpol}) policies in defining controlable model features. Note that the \textit{ALL} project is basically defined as an open ended project with no preset limits on the ranges or values of the model attributes. Hence, the ranges for all the default ranges of the models, defined in \fig{default}.
 
 The lists at the bottom of \fig{ALLall} through to \fig{OSP2tactical}
 show how \eq{ps} 
  ranked
 the various project decisions that, according to STAR and the fitness function expressed in \eq{energy}, achieve the best balance of reducing the various model estimates given the used weights.
 These ranks correspond to the X-axis of the plots at the top of that figure.
 ``Median'' plots the 50\% percentile of the defect/ effort/ schedule/ threat
 values seen
 after imposing ranges $1..X$ (and selecting all other ranges at random).
 ``Spread'' shows the 75\%-50\% percentile range.
 These plots are U-shaped:
 \bi
 \item On the left-hand-side of each plot,
 poor results were seen
 after applying too few constraints.
 That is, models perform poorly if we do not control them enough.
 \item
 On the right-hand-side of each plot, poor results were also seen.
 In terms of \eq{ps}
 , these are the ranges with low support and
 low probability of belonging to $best$ class. Hence, it is not surprising that
 applying these superfluous constraints was counter-productive.
 \ei 
\input{figures/nasa/allFLIGHTresults}
\input{figures/nasa/strategicFLIGHTresults}
\input{figures/nasa/tacticalFLIGHTresults}
 The policies that recommend new ASE tools in the plots are designated by the ``$\blacktriangle$ $\blacktriangle$'' symbol next to them.

Observing the plots and the policies in \fig{ALLall} through to \fig{OSP2tactical}, there are several observations we can make. These are global observations which seem to apply across the board for every one 
\bi
\item
For all the models, except threat, the plots have a U-shape with different degrees if concavity. For these plots, we can also notice that not only are the medians of the estimates being reduced, but that also the spreads are being reduced as well. Also, these spread curve never seem to excessively exceed the medians in any of the project/policy combinations. This implies that through STAR we are able to control the model estimates and consistently reduce them with varying degrees.
\item
The policies achieved automatically in STAR always include a very low percentage of all the possible recommendations in the search space. One example is \fig{ALLtactical}, where the policy contains 12 total recommendations. The search space in this case is about 140 recommendations, making the policy include about 8.6\% of the total space of recommendations. This suggests that STAR is indeed capable of vastly reducing the space of possible recommendations that a manager can chose from.
\item
The initial rapid decrease in the estimate curves, compared to the gradual increase in these curves as superfluous recommendations are added to the list. This suggests that we are able to control, and initially drastically reduce (especially for less constrained projects), the model estimates by controlling a subset of the features in those models. This supports the concept of having ``collar'' features, which in turn supports our approach in selecting a subset of these features to be included in policy recommendations.
\item
The top ranked recommendations produced by STAR always have values that exist on extremum of the range specified by the project. Note that it is this phenomenon that inspired implementing extreme sampling (\tion{staralgo}), as well as SEESAW~\cite{menzies09}.
\ei
\input{figures/nasa/allGROUNDresults}
\input{figures/nasa/strategicGROUNDresults}
\input{figures/nasa/tacticalGROUNDresults}

Aside form the above general observations, we can also comment on the importance of advanced ASE tools. Note that, in all the policies except for one (that is for $\dfrac{14}{15}$ of the cases), advanced ASE tools are being recommended in one form or the other. It should also be noted that, when ASE tools are recommended, they are always recommended at the highest levels available. This results asserts that, not only are these tools important, but that it is also necessary to apply them at their highest levels whenever they are recommended.

Another observation that can be made is that advanced ASE tools become more important as a project becomes more constrained. ``Constrained'' here can be interpreted as more mature and further down the development path, where more of the properties of the projects are buttoned down and set. This can be seen through the policies in the different case studies. One example is how advance ASE tools are ranked $6^{th}$ and $8^{th}$ in \fig{flightstrategic} that presents that results of flight strategic, while they are ranked at $1^{st}$, $2^{nd}$ and $3^{rd}$ in all the OSP2 case studies presented in \fig{OSP2all} to \fig{OSP2tactical}. 

 \input{figures/nasa/allOSPresults}
\input{figures/nasa/strategicOSPresults}
\input{figures/nasa/tacticalOSPresults}

 \input{figures/nasa/allOSP2results}
\input{figures/nasa/strategicOSP2results}
\input{figures/nasa/tacticalOSP2results} 
 
 
\clearpage
In addition to the above figures, we present \fig{nasabraillep1} and \fig{nasabraillep2}. These two figures are what we called braille charts, and aim to summarize the results of the studies in a more concise fashion compared to the above figures. Using these charts, we can make the observation that, for effort and months (development time), the reductions produced for less constrained projects are much larger than those achieved for more constrained ones. In other words, we are able to exploit the uncertainty within these less constrained projects to drive large improvements. This supports the result expressed in ~\cite{me07g}, where it is stated that ``if you fix everything, you have nothing else to fix". Defects reduction, on the other hand, are dramatic for all the cases, with the least reduction brings down defects to 26\% of its initial value, which can be attributed to the use of advanced ASE tools.

\input{tables/braille-nasa-collapsed}
 
Note that these studies seem to show that, even though the threat model seems to make sense theoretically in managing and estimating risk, empirically is seems too insensitive to model inputs, causing low (and usually non-existent) medians and high (and jumpy) spreads (threat never really exceeds 20). The reason behind this is possibly the look-up table nature of this model, where instead of progressive transitions, we have value jumps which are exhibited in a small area of the look-up tables as shown in \fig{madachy}. Due to this behavior, the threat model will not be emphasized throughout this work. Future investigation into redefining or making this model more effective is possible, but is outside the scope of this work.

In conclusion, social factors are indeed very important. However, advanced ASE tools shouldn't be neglected, especially for more defined and constrained projects.
