\documentclass[journal,12pt,onecolumn,compsoc]{IEEEtran}
\newcommand{\aive}{{\"aive}}
%\usepackage{times}
\usepackage{rotating}
\usepackage{wrapfig}
%-- Begin patch area for accents in 'Author Block' area - may be needed by some authors / but not all
%\DeclareFixedFont{\auacc}{OT1}{phv}{m}{n}{12}   % Needed for "Author Block" accents - Patch by Gerry 3/21/07
%\DeclareFixedFont{\afacc}{OT1}{phv}{m}{n}{10}   % Needed for "Author Block" accents in the affiliation/address line - Patch by Gerry 3/21/07
%--
\usepackage{pifont, color,cite,tricks}
\usepackage{url,amssymb}
%\DeclareMathOperator*{\argmin}{arg\,min}
%\newcommand{\hmm}[1]{{\underline {\bf #1}}}
%\usepackage{eso-pic}
%\usepackage{color}
%\usepackage{type1cm}
% \makeatletter
%   \AddToShipoutPicture{%
%     \setlength{\@tempdimb}{.5\paperwidth}%
%     \setlength{\@tempdimc}{.5\paperheight}%
%     \setlength{\unitlength}{1pt}%
%       \makebox(600,900){\rotatebox{45}{\textcolor[gray]{0.85}{\fontsize{9cm}{9cm}\selectfont{Draft}}}}}
% \makeatother
\begin{document}
%
% --- Author Metadata here ---
%\CopyrightYear{2007} % Allows default copyright year (200X) to be over-ridden - IF NEED BE.
%\crdata{0-12345-67-8/90/01}  % Allows default copyright data (0-89791-88-6/97/05) to be over-ridden - IF NEED BE.
% --- End of Author Metadata ---

\title{Future Trends in the General SE Industry\\
part 2: Simulation Results}
% of EIGHT authors. SIX appear on the 'first-page' (for formatting
% reasons) and the remaining two appear in the \additionalauthors section.
%
% You can go ahead and credit any number of authors here,
% e.g. one 'row of three' or two rows (consisting of one row of three
% and a second row of one, two or three).
%
% The command \alignauthor (no curly braces needed) should
% precede each author name, affiliation/snail-mail address and
% e-mail address. Additionally, tag each line of
% affiliation/address with \affaddr, and tag the
% e-mail address with \email.
%
% 1st. author
\author{Tim Menzies, Barry Boehm\thanks{%
Dr. Menzies is with the Lane Department of Computer Science and Electrical Engineering,
West Virginia University, USA; \protect\url{tim@menzies.us}.}\thanks{%
Dr. Boehm is with the University of Southern California,
USC Center for Software Engineering.
\protect\url{boehm@sunset.usc.edu}}\thanks{
This research was conducted at West Virginian University and the University of Southern California
under a SARP NASA grant ``Crysal Ball''.
Reference herein to any specific commercial product, process, or service 
by trade name, trademark, manufacturer, or otherwise, does not constitute or imply 
its endorsement by the United States Government.
} \\\today}

      % There's nothing stopping you putting the seventh, eighth, etc.
% author on the opening page (as the 'third row') but we ask,
% for anesthetic reasons that you place these 'additional authors'
% in the \additional authors block, viz.
% Just remember to make sure that the TOTAL number of authors
% is the number that will appear on the first page PLUS the
% number that will appear in the \additionalauthors section.
\date{today}
\maketitle
\begin{abstract}
%Adoption of advanced automated SE (ASE) tools would be more favored if a                                   
%business case could be made that
%these tools are
%more valuable than alternate methods.
%In theory, software prediction models can be used to make that case.
%In practice, this is complicated by the
%"local tuning" problem. Normally, predictors for software effort
%and defects and threat use local data to tune their predictions.
%Such local tuning data is often unavailable.

%This paper shows that assessing the relative merits of different
%SE methods need {\em not} require   precise local tunings.  STAR1 is a
%simulated annealer plus a Bayesian post-processor that explores
%{\em the space of possible local tunings} within software prediction models.
%STAR1 ranks project decisions by their effects on effort
%and defects and threats.
%In experiments with NASA systems, STAR1 found one project
%where ASE were {\em essential} for minimizing effort/ defect/ threats;
%and another project were ASE tools were merely
%{\em optional}.

Adapting SE practices and methods to future software trends is important in an
industry that is constantly changing and evolving. In order to do so effectively,
two high level steps need to be taken. The first step is determining what future
software trends will look like. This can not be done in a determinant fashion, and
hence is done by collecting educated guesses from seasoned software engineering experts. 

A prior version of this document
described a set of trends identified by an expert panel especially convened at ICSE'08
to discuss this issue. This document extends that prior document with
results from the simulation of those trends. The first half of this
document comes from the prior document while the second half
contains new material.
\end{abstract}
\newpage
\tableofcontents
\newpage
\listoffigures
\newpage
% A category with the (minimum) three required fields
%\category{I.6}{Learning}{Machine Learning}
%\category{D.2.9}{Software Engineering}{Effort Estimation}
%A category including the fourth, optional field follows...
%\category{D.2.8}{Software Engineering}{Metrics}[product metrics, process metrics]


%\keywords{COCOMO, COQUALMO,  simulated annealing, Bayes}
\pagestyle{plain}
\section{Introduction}\label{sec:intro}


Here are some hard questions:
\bi
\item
	Where should the NASA agency be, in (2,5,10,20) years times? 
\item	At that time, what practices would ensure the success of NASA missions? 
\item	Is the agency currently conducting those practices? 
\item	If not, what changes are required to current practices to assure future NASA systems?
\ei

These are very current questions. Much is changing within the agency.
For example, a recent new initiative at NASA IV\&V is the use of
early life cycle model-based validation. Elsewhere in the agency,
there is increasing interest in agile process, assertion-based
analysis,  Eclipse-based programming, Matlab-based automatic code
generation, simulation-oriented development cycles, etc. How are
we to assess the benefits vs cost trade-offs on all these new
methods?  How are we to make future plans for the agency, given
some much change in current practices?

Using traditional methods, there are no answers to these questions.
Such a study is hindered by the {\em local tuning} problem. Software
process models are most accurate after they have been tuned to local
data. Unfortunately, the data required for local tuning is difficult
to obtain, especially across multiple organizations . This  is due
to the business sensitivity associated with the data as well as
differences in how the metrics are defined, collected and archived.
Sometimes, the data is not even archived (e.g. after two years we
were only able to add 7 records to our  NASA wide software cost
metrics repository).

Recently, we have developed a novel solution to the local tuning
problem. We have found that, in terms of ranking different approaches
to software development, a precise local tuning is not required.
Rather, some {\em knowledge of local circumstances} may suffice to
make the business  case for some policy (A,B,C) versus some alternate
policy (X,Y,Z)~\cite{me07f,me07g,me08c}. Based on research dating
back to 1981~\cite{boehm91,boehm00b},  we  assert that the space
of possible tunings seen in actual practice is well known.  This
space can be explored to find features that minimize effort, defects,
development time, and other threats to the development process.
While we can not say exactly how much some policy (A and B and C)
will change effort, defects, and development time, we can comparatively
assess policy (A,B,C) against some other policy (X,Y,Z). Better
yet, we can explore a very large space of different policies.

In this paper, we explore this kind of analysis over a space of
projected changes to general software engineering, with a particular
focus on NASA-like applications.  In particular:
\bi
\item We define the input
space for a class of applications relevant to NASA projects.
\item We show the effects of running those ranges though our
AI optimizer
\ei

Note one minor clarification before we proceed.  As
mentioned in the prior version of this document,
the  title of this report
is ``Future trends in the general SE industry'' since that is one of the
named deliverables in the project plan accepted by the sponsors of this work (NASA).  
That title is slightly
misleading since, as we shall see, our expert panel found that topic too broad.
In order to make specific conclusions, the panel recommending focusing in on
a particular class of system that was both common in the general industry and
at NASA. Hence, a more accurate (but overly verbose)
title for this report might be ``Future trends in 
specialized sections of the general SE industry relevant to NASA''.
\newpage
\input{cocomo}
\section{Models}

Our panel expressed their views on the future of software engineering in terms of the USC COCOMO-family
models:
\bi
\item
The COQUALMO defect predictor~\cite[p254-268]{boehm00b};
\item
The  COCOMO effort predictor~\cite[p29-57]{boehm00b}; 
\item
The THREAT  predictor for effort \& schedule overrun~\cite[284-291]{boehm00b}.
\item
The COCOMO schedule predictor for development time~\cite{boehm00b}.
\ei

Hence, before discussing the panel's comments, we need first review those models.
Such a definition can be achieved via the ontology of COCOMO, COQUALMO, and THREAT, shown in \fig{cparems}.
This figure lists a variety of project {\em features} with the 
{\em range}
\{very low, low, nominal, high, very high, extremely high\} or
\[\{vl=1, l=2, n=3, h=4, vh=5, xh=6\}\]

\subsection{Effort}
The COCOMO effort predictors estimates  development effort in terms of person months where one month is 152
hours (and includes development and management hours). 
In COCOMO, the  {\em scale factors} $SF_i$ of \fig{cparems}
effect effort exponentially (on KSLOC, i.e. Thousand Source Lines of Code)
while {\em effort multipliers} $EM_j$
effect effort linearly:

{\small\begin{equation}\label{eq:coc2}
PM=a*\left(KSLOC^{\left(b+0.01*\sum_{i=1}^{5}SF_i\right)}\right)*\left(\prod_{j=1}^{17}EM_j\right)
\end{equation}}
where KSLOC is estimated
directly or computed from a function point analysis;
 $SF_i$ and $EM_j$ are the {\em scale factors};
{\em effort multipliers} of \fig{cparems};
and
$a$ and $b$ are  domain-specific parameters. In our NASA data,
these ranges\begin{equation}\label{eq:ab}
\left( 3.72 \le a \le 9.18 \right) \wedge \left(0.88 \le b \le 1.09\right) 
\end{equation} 
With the effort multipliers, off-nominal ranges (i.e. \{vl=1, l=2, h=4, vh=5, xh=6\}) change the prediction by some amount.
In the special case of the nominal range (i.e. \{n=3\}), that amount is one; i.e. the nominal
range make no change to the prediction.
Hence, these  ranges  can be modeled as a straight line $y=mx+b$
passing through the point $\{x,y\}$=$\{3,1\}$. Such a line has a y-intercept
of $b=1 - 3m$. Substituting this value of $b$ into $y=mx+b$ yields:
\begin{equation}\label{eq:lineem}\small
\begin{array}{l}
\forall x \in \{1..6\}\; EM_i=m_a(x-3) + 1
\end{array}
\end{equation}
where  $m_a$ denotes the effect of effort multiplier $a$ on {\em effort}.

The effort multipliers form
into two sets:
\be
\item
The {\em positive effort EM} features, with slopes $m^+_a$, that are proportional to effort. 
These features are:
cplx,
data,
docu,
pvol,
rely,
ruse,
stor, and
time.
\item
The
{\em negative effort EM} features, with slopes $m^-_{a}$,
are inversely proportional to effort.
These features are 
acap,
apex,
ltex,
pcap,
pcon,
plex,
sced ,
site,
tool.
\ee
Based on prior work~\cite{boehm00a}, we can describe
the
space of known tunings for COCOMO effort multipliers to be 
\begin{equation}\label{eq:mem}\small
\left(0.073 \le m^+_a \le 0.21\right) \wedge
\left(-0.178 \le m^-_a \le -0.078\right)
\end{equation}
Similarly, using experience from 161 projects~\cite{boehm00a},
we can say that the space of known tunings for the COCOMO scale factors 
(prec, flex, resl,team, pmat)
are:
\begin{equation}\label{eq:linesf}\small
 \forall x \in \{1..6\}\; SF_i=m_b(x - 6) \wedge \left(
-1.56	\le m_b \le -1.014\right)
\end{equation}
where  $m_b$ denotes the effect of scale factor $b$ on {\em effort}.

Note that the above ranges for the slopes were obtained by finding the average slope for each COCOMO attribute for both effort multipliers and scale factors over the range of values of that attribute.

\subsection{Schedule }
The COCOMO schedule predictors estimates  development time in terms of elapsed calendar months.
This model is largely based on the COCOMO effort in that it uses
that attributes, there values, and the calibrated coefficients (i.e, $a$ and $b$).
The following are the equations for calculating months:

{\small\begin{equation}\label{eq:mon}
\begin{array}{c}
F=d+0.2*\left(0.01*(\sum_{i=1}^{5}SF_i)\right)\\
TDEV=\left[c*(PM_{NS})^{F}\right]*SCED\%\div100
\end{array}
\end{equation}}
where $PM_{NS}$ is the effort calculated without the inclusion of SCED, 
$SF_i$ are the {\em scale factors} of \fig{cparems},
$c$ and $d$ are  domain-specific parameters that can be calibrated,
and $SCED\%$ is the percentage of schedule relative to the nominal 
schedule. 

\fig{sced} shows the value of this parameter based on 
SCED.$TDEV$ is the development time in months calculated by this model,
and $F$ is referred to as the scaling exponent for schedule.

\begin{figure}[!h]
{\scriptsize
\begin{center}
\begin{tabular}{c|c}
SCED&SCED\%\\\hline
1&75\\
2&85\\
3&100\\
4&130\\
5&160\\
\end{tabular}
\end{center}
} \caption{The values of SCED\%}\label{fig:sced}
\end{figure}

\subsection{Defects} 
COQUALMO has two core models, used three ways:
\bi
\item
 The {\em defect introduction} model is similar to
\eq{coc2}; i.e. settings to \fig{cparems}'s effort multipliers and scale factors
map to predictions about number of defects.
\item
The {\em defect removal} model represents how various tasks (peer review, execution-based
testing, and automatic analysis) decrease the introduction of defects.
\item
The above two models are repeated three times 
for
defects introduction \&  removal for
requirements, design, or coding.
\ei
COQUALMO follows the same convention as COCOMO for the effort multipliers; i.e.
nominal values ($n=3$) add nothing to the predicted number of defects. As
above, COQUALMO is:
\begin{equation}\label{eq:linedefintro}\small
\begin{array}{l}
 \forall x \in \{1..6\}\; EM_i=m_c(x-3) + 1
\end{array}
\end{equation}
where  $m_c$ denotes the effect of $c$ on {\em defect introduction}.

The effort multipliers and  scale factors form two sets:
\be
\item The {\em positive defect} features, with slopes $m^+_c$, that are proportional
to the estimated introduced defects. These features are 
flex,
data,
ruse,
cplx,
time,
stor, and 
pvol.
\item The {\em negative defect} features, with slopes $m^-_c$, that are inversely proportional
to the estimated introduced defects. These features are 
acap,
pcap,
pcon,
apex,
plex,
ltex,
tool,
site,
sced,
 prec,
resl,
team,
pmat,
rely, and
docu.
\ee
The space of tunings for  defect introducing features are:
\begin{equation}\label{eq:coq}\small
\begin{array}{c}
requirements\left\{
\begin{array}{c}
0 \le m^+_c \le 0.112\\
-0.183 \le m^-_c \le -0.035
\end{array}
\right. \\
~\\
design \left\{
\begin{array}{c}
0 \le m^+_c \le 0.14\\
-0.208 \le m^-_c \le -0.048
\end{array}
\right. \\
~\\
coding \left\{
\begin{array}{c}
0 \le m^+_c \le  0.140\\
-0.19 \le m^-_c \le -0.053
\end{array}
\right.
\end{array}
\end{equation}

\noindent
The space of tunings for  defect removal features are:
\begin{equation}\label{eq:linedefrem}\small
\begin{array}{rl}
\forall x \in \{1..6\} &SF_i=m_d(x - 1)\\
requirements: 	& 0.08 	\le m_d \le 0.14 \\
design: 		& 0.1 	\le m_d \le 0.156 \\
coding: 		& 0.11 	\le m_d \le 0.176 \\
\end{array}
\end{equation}
where  $m_d$ denotes the effect of  $d$ on {\em defect removal}.

\subsection{Effort \& Schedule Overrun}\label{sec:threat}

The THREAT model
returns a heuristic estimate of the threat of a
schedule over run in the project. 
This estimation model is dependent upon
the COCOMO effort multipliers

Internally, THREAT contains dozens of tables of the
form of \fig{madeg}. Each such table  adds some ``threat'' value
to the overall project risk when multiplied by
the effort multiplier values of the corresponding COCOMO attributes.
 There are six major categories: schedule, product, personnel, process, platform and reuse.
After the threat for each category is calculated, the sum is
normalized to produce the final threat rating.


\begin{figure}[!t]
{\scriptsize
\begin{center}
\begin{tabular}{|p{0.8in}|c|c|c|c|c|}\cline{2-6}
\multicolumn{1}{c|}{} &rely=&rely=&rely=&rely=&rely=\\
\multicolumn{1}{c|}{}& very & low &nominal & high & very\\
\multicolumn{1}{c|}{}&  low&  & &  &  high\\\hline sced= very
low& 0 & 0 & 0  &1  &2\\\hline sced= low     & 0 &0  & 0 &
0&1\\\hline sced= nominal& 0 & 0 & 0  &0  &0\\\hline sced= high&
0& 0 & 0  & 0 &0\\\hline sced= very high& 0 & 0 & 0  &0 &0\\\hline
\end{tabular}
\end{center}
} \caption{An example risk table}\label{fig:madeg}
\end{figure}

\fig{madeg} can be represented as
an exponentially decaying function that peaks in one corner of the risk table
at a value of two.
All the tables peak at either a value of two  or four.
Since this model is heuristic in nature, the exact height of the peak is
not certain.
When we perform Monte Carlo simulations over THREAT, we vary the height of
the peak by 
a random factor $0.5 \le x \le 1$ if the peak is four, and $0.5 \le x \le 1.5$ if the peak is two.


\newpage
\section{At the Panel}

\subsection{Methodology}
Our data was collected as followed:
\bi
\item Using their contacts in the research community, the authors invited a set of seasoned
software engineering researchers to an evening panel at the 2008 International Conference on
Software Engineering. 

\item
Prior to the panel, experts were sent briefing charts on the project and the kinds of inputs needed
by the tools. In turned out those notes were less-than-optimal and the first 30 minutes of the panel
was used explaining the way our AI tools can search a space of options.
\item
Next, the panel focused their discussion on  a particular class of software. This focus was necessary,
said the panel, in order to crystallize the discussion. After some debate, the panel focused on the
software describes in \fig{focus}. This particular class of software was chosen since it
\be
\item Represents a common kind of system seen in industry;
\item It also represents a common kind of system seen at NASA.
\ee
\begin{figure}[!b]
\begin{center}
\begin{tabular}{|p{4in}|}\hline
In order to make specific conclusions, the panel recommending focusing in on
a particular class of system that was both common in the general industry and
at NASA. 

Accordingly, this study explored:
\bi
\item A mission critical, real-time system;
\item Built by contractors (not in-house)
\item That has an operational life of 5 to 10 years (since have invested much effort into a mission critical
system, an organization is most likely to use it for many years to come).
\ei
\\\hline
\end{tabular}
\end{center}
\caption{System used in this study.}\label{fig:focus}
\end{figure}
\item
Next, each participants was given
a table with two columns. The left hand column listed the features of \fig{cparems} and the right hand column
was blank.
The participants were asked to to discuss on the expected value of each feature in 10 years time.
\item 
In order to assure that the panel shared a common definition of the features, before discussing a feature,
a COCOMO authority (Boehm) gave a 3 minute lecture on the meaning of each feature.
\item
This was the first time this kind of knowledge acquisition exercise was conducted for our tools.
Hence, to allow for feedback from the participants
to shape the process,
there was very little initial focus offered by the moderators. 
\item
It proved difficult to get precise and exact agreement on the future trends of these features is difficulty.
45 minutes later, we were still discussing the first feature.
For this reason, the knowledge acquisition method was changed.
\item
For 2 hours, the panel offered comments on future trends in SE as follows.
After brief discussions about every attribute, a vote was taken concerning the {\em qualitative
future behavior of the feature}. \ei
\newpage
These {\em qualitative future behaviors} offered by the panel had one of four shapes:
\bi
\item
\textbf{Up}: The attribute would increase in value. For example, ACAP was voted to have such behavior.
\item
\textbf{Down}: The attribute would decrease in value. APEX was voted to have such behavior.
\item
\textbf{Flat}: No change from the baseline.
\item
\textbf{Bump}: The feature is bi-modal and can  have one of two values: either {\em up} or {\em down} (described above).
\item
\textbf{Rick}: Some panelists wanted another shape were the expected value was usually low but may be
sometimes quite high.
This shape proved to only be used for features were the the panel had no consensus view. Hence, we do not explore
it further.
\item
\textbf{Abstain}: Some chose to abstain from some of the votes as well. 
The number of abstains are shown in Figures 5 \& 6.
\ei
\subsection{Results}
For full results, see Figures 5 \& 6.
For a quick summary, see below.
\subsubsection{No Consensus}
The trend votes given to all features by the panel are shown in the appendix.
When features had no consensus, we made no comment. Those features, which
we will ignore for the rest of this report, were:
\bi
\item data (database size per LOC)
\item ett (execution-based testing and tools)
\item reuse (level of design of the current product for future reuse)
\item time (\% CPU requirements)
\ei
\subsubsection{No Change}
The panel believed that no change will be observed in the following features:
\bi
\item docu (required levels of documentation)
\item ltex (language and tools experience)
\item pcap (programmer capability)
\item plex (platform experience)
\item team (team cohesion)
\ei
\subsubsection{Increasing}
The panel believed that the following trends will increase:
\bi
\item aa (automated analysis): i.e. use of more automated analysis;
\item acap (analyst capability): due to better SE education, broader international talent pool;
\item cplx (product complexity): i.e. due to more ambitious projects;
\item flex (development flexibility): due to more agile-style development;
\item peer (peer reviews).
\item pmat (process maturity)
\item pvol (platform volatility)
\item rely (reliability)
\item resl (architecture and risk analysis)
\item tool (use of software tools)
\ei
\subsubsection{Decreasing}
The panel believed that the following trends will decrease:
\bi
\item apex (analyst experience): due to a wider variety of new tools and 
an increasing number of novel application areas
\item pcon (personnel continuity): i.e. the panel expects
{\em more} turn over in the industry\footnote{Note that $pcon$ is defined
in $reverse$ where lower levels mean higher turnover.}.
\item sced (time to deliver product): i.e. more products expected to be
delivered faster\footnote{Note that, like $pcon$, the $sced$ feature
is defined such that lower levels mean more requirements to deliver faster}.
\item site (single site development): i.e. less work at one location,
more development at multiple distributed locations;
\ei
\subsubsection{Bi-modal}
The panel believed that the following trends are bimodal; i.e. will increase
or decrease for different applications.
\bi
\item prec (have we built this kind of thing before?)
\item stor (\%CPU RAM)
\item time (\%CPU cycles)
\ei
\subsubsection{Sanity Checks}
From the above, we note certain contradictions that will have to be checked
in subsequent panels:
\bi
\item plex=steady (platform experience) but increased pvol (platform volatility).
\ei
\input{icseMay13}
\clearpage
\section{Simulations}
\begin{wrapfigure}{r}{2in}
\begin{center}
\scriptsize
\begin{tabular}{l|c|l|c}
feature&setting&feature&setting\\\hline
aa&3&tool&3\\
acap&3&time&3\\
apex&4.5&team&3.5\\
cplx&3&stor&3\\
data&4&site&4.5\\
docu&3&sced&3\\
ett&2&ruse&4\\
flex&3&resl&3\\
ltex&4.5&rely&2.5\\
pcap&3&pvol&2\\
pcon&3&prec&4.5\\
peer&2.5&pmat&2.5\\
plex&4.5\\
\end{tabular}
\end{center}
\caption{Baseline}\label{fig:base}
\end{wrapfigure}
Our simulations seek consistent trends and patterns
in project improvement for \fig{focus}.

Those simulations are run as follows. 
If
there was consensus about each trend in the panel, 
then our simulations bias the runs according to the trends.
Else, our simulations do not bias that variable and the simulation  tries all values in that range.
Also, we run our simulations twice for each bump: once assuming {\em up} and
once assuming {\em down}.
\bi
\item
For features trending {\em up}, then we allow the simulation to select any input {\em at and above} known
baselines.
\item
For features trending {\em down}, then we allow the simulation to select any input {\em at and below} known
baselines.
\item For features trending {\em flat}, then we just use the baseline
figure, without change.
\item
\fig{base}, built from Boehm's knowledge, will be   our baseline.
\ei
When search for changes from the baseline, we explore in two modes:
strategic and tactical.
The last two columns of \fig{emsf2} 
show the results of  prior expert panel session at JPL where the COCOMO variables
were separated into those $tactical$ variables
that can be changed within the space of one project,
and those $strategic$ variables
that required higher-level institutional change (and so may take
longer to change). For example, the panel declared that $pmat$ (process maturity)
is hard to change within the space of a single JPL project. 
In the sequel, all our RANKER experiments will be divided into those that
just use the $strategic$ variables and those that just use the $tactical$
variables.
\begin{figure}[!h]
\begin{center}
{\scriptsize
\begin{tabular}{p{3cm}|r@{:~}l|c|c|}
\multicolumn{3}{c}{~}&strategic?&\multicolumn{1}{|c}{tactical?}\\\cline{2-5}
scale   &prec & have we done this before?&\ding{51}&\\
factors &flex & development flexibility &&\ding{51}\\
(exponentially        &resl & any risk resolution activities?&&\ding{51}\\ 
decrease       &team &  team cohesion&&\ding{51}\\ 
effort)       &pmat & process maturity&\ding{51}&\\\hline 
upper  &acap & analyst capability&\ding{51}&\\
(linearly       &pcap & programmer capability&\ding{51}&\\ 
decrease      &pcon & programmer continuity&\ding{51}&\\
effort)       &aexp &  analyst experience&\ding{51}&\\       
&pexp &  programmer experience&\ding{51}&\\       
&ltex &  language and tool experience&\ding{51}&\\       
&tool &  tool use&&\ding{51}\\       
&site &  multiple site development&\ding{51}&\\       
&sced & length of schedule  &&\ding{51} \\\hline 
lower &rely &    required reliability && \\
(linearly      &data &   secondary memory  storage requirements&&\ding{51}\\
increase      &cplx &  program complexity&&\ding{51}\\
effort)      &ruse &  software reuse&&\ding{51}\\      
&docu &   documentation requirements&&\ding{51}\\      
&time &   runtime pressure&&\\      
&stor &   main memory requirements&&\ding{51}\\     
&pvol &    platform volatility  &&\\\hline
COQUALMO &auto& automated analysis&\ding{51}&\ding{51}\\
defect removal&ett & execution-based testing tools&\ding{51}&\ding{51}\\
methods&peer& peer reviews&\ding{51}&\ding{51}\\\cline{2-5}
\end{tabular}}
\end{center}
\caption{Strategic \& tactical controllables.}\label{fig:emsf2}
\end{figure}

Note that these definitions of $strategic$ and $tactical$ 
are not hard-wired into STAR and can be changed
via editing a  
simple configuration file.

\subsection{Simulated Annealing}
{\em Monte Carlo} algorithms randomly sample the space
of possible controllable model states. 
A  {\em Metropolis} Monte Carlo algorithm~\cite{metropolis53}
creates new states
by  small mutations to some
{\em current} state. 
If a new state is ``better'' (as assessed via an
{\em energy function}),  it becomes the new {\em current}
state used  for future
mutations.  Otherwise, a Boltzmann acceptance
criteria is used to probabilistically   decide to
assess the new state: the worse the new state, the less
likely that it becomes the new current state.
The algorithm is silent on the mutation mechanism. For our experiments,
we freeze $\frac{2}{3}$ of the features and
randomly  select ranges for the rest.

In 1983, Kirkpatrick et.el.~\cite{kirkpatrick83} 
proposed a modification that was
inspired
by a procedure used to make the strongest possible glass. Initially,
glass is heated to allow atoms to move freely. The temperature
is then slowly lowered such that the atoms can find the stablest
configuration with lowest energy. 
A {\em simulated annealing} (SA) algorithm adds
a ``temperature'' variable to the Boltzmann accept criteria
such that, at high temperatures, it is more likely that the
algorithm will jump to a  new worst current state.
This allows the algorithm to jump out of local minima while
sampling the space of options. As the temperature cools,
such jumps become less likely and the algorithm reverts
to a simple hill climber.

With SA, better solutions have lower ``energy''
so the fitness function $E$ is defined such that $E \ge 0$ and
{\em lower} values are
$better$. 
For our purposes, we model effort ($Ef$) as the distance to
the origin of the 4-D space with defects ($De$), threats ($Th$), and schedule ($Mo$):
\begin{equation}\small\label{eq:eng}
\begin{array}{c}
RD = relydefect^{(RELY-3)}\\\\
E=\frac{\left(\sqrt{(\overline{Ef}*\alpha)^2 + (\overline{De}*(\beta+RD))^2 + (\overline{Th}*\gamma)^2 + (\overline{Mo}*\delta)^2}\right)}{\sqrt{\alpha^2 + (\beta+RD)^2 + \gamma^2 + \delta^2}}\\
\end{array}
\end{equation}
Here, $\overline{x}$ is a normalized value 
$0 \le \frac{x - min(x)}{max(x) - min(x)} \le 1$. 
Hence, our energy ranges $0 \le E \le 1$ and
{\em lower} energies are {\em better}. $\alpha$, $\beta$, $\gamma$, and $\delta$ 
are the weighting factors for the different models. $relydefect$ is a constant
that defines a relationship between RELY and the defect model: as RELY increases,
the weighting of the defect model increases as well.

The acceptance criteria $P$ for a new state is defined using
the current temperature $T$, 
the energy of the current solution ($e$),
and the energy of the new mutation ($en$):
\begin{equation}\small
P(e,en,T) = e^{(e -en)/T}
\end{equation}
$T$ is defined to $decrease$ as the simulator loops through $k= 1\ldots kmax$ iterations.
We use
$T = e^{-100*k/kmax}$.

\begin{figure}[!t]
\scriptsize
\begin{verbatim}
function sa(kmax) 
  s := s0; e := E(s)           // Initial state, energy.
  sb := s; eb := e             // Initial "best" solution
  k := 0                       // Energy evaluation count.
  while k < kmax and e > emax  // Loop
    sn := neighbour(s)         //   Pick some neighbour.
    en := E(sn)                //   Compute its energy.
    if en < eb then            //   Is this a new best?
      sb := sn; eb := en       //     Yes, save it.
    if random() < P(e, en, temp(k/kmax)) 
     then s := sn; e := en     //     Maybe jump
    k := k + 1                 //   One more evaluation done
  return sb                    // Return best
\end{verbatim}
\caption[SA pseudo-code]{SA pseudo-code:
a new solution $sn$ (with new energy $en$) replaces the current solution                                  
if (a)~it has a lower energy or (b)~the acceptance predicate $P$
endorses it. Only in the case of (a)
should the new solution
replaces the current best solution.
} \label{fig:sa}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[width=2.5in]{jpl1.pdf}
%eps/JPL.pdf}
\end{center}
\caption{Processing a JPL requirements model. }\label{fig:jpl}
\end{figure}
Two advantages of SA are their {\em implementation simplicity} and their {\em ability to handle
non-linear models}:
\bi
\item {\em Implementation simplicity:}
\fig{sa} illustrates the simplicity of the algorithm. Memory is only required
for one current solution ($s$), one new solution ($sn$) and one best solution ($sb$) that stores
the best solution seen at any time in the simulation.
\item
{\em Non-linear models:}
 Previously~\cite{fea02a,me03i}, we have applied SA to non-linear JPL requirements models
where minimizing the cost of project mitigations can decrease the number of requirements
achieved by that project. Hence, decreasing {\em both} the cost and achieved requirements
is a non-linear problem that must trade between minimizing cost and increasing
requirements coverage.
The top-left line  of \fig{jpl} divides the behavior of the JPL requirements models
before  and after  simulated annealing. 
As shown below the line, 
initial Monte Carlo sampling of the possible mitigations lead to a large range
of costs and benefits. Simulated annealing found a set of mitigations
that lead to the small cloud of solutions above the line. Compared to the initial samples, these
new solutions had decreased cost, increased benefit (number of requirements covered), and decreased variance (shrank
the space of solutions). 
\ei
Two disadvantages of SA algorithms are their {\em incompleteness} and the {\em
complexity}
of their solutions:
\bi
\item 
{\em Incompleteness:} In our domain, we have some evidence that the incomplete nature of the heuristic SA
search is not a major problem.
\fig{saeg} shows a sample run of our SA tool running on our prediction models
for $K=10,000$ simulations. 
As $k$ increases for $1{\ldots}10,000$, it becomes less and less
likely that a better $best$ has been missed. Hence, we run our simulations
for ten times the period it takes for $best$ to stabilize (at $k\approx 1000$). 
\begin{figure}
\begin{center}
\includegraphics[width=2.7in]{saeg1.pdf}
\end{center}
\caption[SA output]{Dots \& lines are
SA output from current \& best solution (respectively)  after $k$ simulations.
}\label{fig:saeg}
\end{figure}
\item
{\em Solution complexity:}
Simulated annealers offer constraints to all controllable features. 
Often this
is an over-constrained solution since, 
in many domains, a repeated empirical result is a {\em feature subset selection}
effect; i.e. models that constrain M variables perform just as well, or better,
than models that constrain N variables ($M\ll N$)~\cite{kohavi97,hall03}.
For example, Kohavi~\cite{kohavi97}
studied some machine learners to find  that using just 19\% of the available features
{\em increased} prediction accuracy by just 2.14\% (on average).
For another example,
when feature subset selection was applied to the JPL requirements model of \fig{jpl}, we found that
up to $\frac{2}{3}$-rds of the features can be left unconstrained,
without effecting the conclusions~\cite{fea02a}. 
\ei
%
%\begin{figure}[!t]
%\scriptsize
%\begin{center}
%\begin{tabular}{|l|r|r|r|p{0.95cm}|}\cline{2-4}
%\multicolumn{1}{c}{}     & \multicolumn{3}{|c|}{av.  number of features} \\\hline    
%dataset         &before &after &$\frac{after}{before}\%$ &accuracy change\\\hline    
%breast cancer &10  &2.9 &29\% &+0.14\%   \\\hline    
%cleve           &13  &2.6 &2\%  &+5.89\%   \\\hline 
%   crx             &15  &2.9 &19\% &+4.49\%   \\\hline  
%  DNA             &180 &11  &6\% &+3.63\% \\\hline    
%horse-colic     &22  &2.8 &13\% &+1.63\%   \\\hline
% Pima            &8   &1   &13\% &+0.79\%   \\\hline   
% sick-euthyroid &25  &4   &16\% &+0.38\%   \\\hline    
%soybean         &35  &12.7 &36\% &+0.15\%  \\\hline\hline    
%average         &38.5&4.99 &19\% &+2.14\% \\\hline
%\end{tabular}
%\end{center}
%\caption{Feature subset selection results from~\cite{kohavi97}}\label{fig:fssr}
%\end{figure}

In terms of  the goal of this paper, exploring feature subset selection
is very important.  Before we can {\em remove} non-essential features,
we must first  {\em rank} them according to their effectiveness.
As we shall see, this ranking will be insightful to the task of
assessing the relative value of new
 ASE
tools and other methods
for reducing effort/ defects/ threats.

\subsection{Ranking Methods}


STAR2 ranks the features ranges
seen in $K$ runs of a simulated annealer by dividing the $K$ runs into:
\bi
\item {\em Best:} those associated with the  BEST\% solutions
(i.e. those with the BEST\% {\em least} energy); 
\item And the {\em rest} (i.e. the other 100-BEST\% of solutions).
\ei
It then computes the probability that a range is found in $best$.
using Bayes' Theorem.
Informally,
the
theorem says $next=old*new$ i.e. what we'll believe {\em next}
comes
from how {\em new} evidence effects {\em old} beliefs.  More
formally:\begin{equation}\label{eq:n}\small
P(H|E)=P(E|H) P(H)\; /\; P(E)\end{equation}
i.e. using
evidence $E$ and a prior probability $P(H)$ for
hypothesis $H\in\{best, rest\}$.
The theorem calculate a  posteriori  probability
$P(H|E)$. 
Simple  Bayes classifiers are often called ``n\aive'' since they assume
independence of each feature. While this assumption
 simplifies the implementation  
(frequency
counts are required only for each feature),
 it is possible
that
correlated events are missed by this ``n\aive'' approach.
Domingos and Pazzini show theoretically
that the independence assumption is a problem in a vanishingly small percent
of cases~\cite{domingos97optimality}. This explains the repeated
empirical result that, on average, seemingly n{\aive} Bayes classifiers perform as well
as other seemingly more sophisticated schemes (e.g. see Table~1 in~\cite{domingos97optimality}).


When applying the theorem, {\em likelihoods}
are computed from observed
frequencies, then normalize to create
probabilities (this normalization cancels out $P(E)$ in \eq{n}, so it need not be computed).
For example, after $K=10,000$ runs divided into  1,000 lowest 10\% {\em best} solutions and 9,000
{\em rest}, the  range $rely=vh$ might appears 10 times in the $best$ solutions, but only
5 times in the $rest$. Hence:

{\scriptsize
\begin{eqnarray}\nonumber
E&=& (reply=vh)\\\nonumber
P(best)&=&1000/10000 = 0.1\\\nonumber
P(rest)&=&9000/10000 = 0.9\\\nonumber
freq(E|best)&=&10/1000 = 0.01\\\nonumber
freq(E|rest)&=&5/9000 = 0.00056\\\nonumber
like(best|E)&=&freq(E|best) \cdot P(best)=0.001\\\nonumber
like(rest|E)&=&freq(E|rest) \cdot P(rest)= 0.000504\\\label{eq:ten}
P(best|E)&=& \frac{like(best|E)}{like(best|E) + like(rest|E)}=0.66
\end{eqnarray}}
Previously~\cite{clark05}
we have found that  \eq{ten} 
is a poor ranking heuristic
since it is distracted by low frequency evidence.
For example, note how
the probability of $E$ belonging to the best class
is moderately high even though its support is very low; i.e. $P(best|E)=0.66$ but
$freq(E|best) = 0.01$. 

To avoid such unreliable low frequency evidence, we augment \eq{ten}
with a support term.
Support should $increase$ as the frequency of a range $increases$,  i.e. 
$like(x|best)$ is a valid support measure.
STAR2 hence ranks ranges via
\begin{equation}\label{eq:ps}\small
P(best|E) * support(best|E) = \frac{like(x|best)^2}{like(x|best) + like(x|rest)}
\end{equation}

\subsection{The STAR2 Algorithm}
To apply \eq{ps}, STAR2 runs in six phases.
In terms of standard machine 
learning theory, step 1 generates a training set; steps 2,3,4 do some generalization;
and step 5 tests the learned theory on data not seen during training.

\begin{description}
\item[1. {\em SAMPLE:}]~\newline
To sample the ranges from the models,
STAR2 runs the simulated annealer  $K_1$ times.
\item[2. {\em DISCRETIZE:}]~\newline
The data seen in the $K_1$ samples is then discretized into $D=10$ bins.
Discretization converts a continuous range into a histogram
with $n$ break points $b_1\ldots b_n$ where $\left(\forall i < j : b_i \le b_j\right)$. 
After discretization,
many observations can fall into the same range
between $b_i$ and $b_{i+1}$
at frequency counts $c_i$. This study used equal width discretization; i.e. 
\[\forall i,j:
\left(b_i - b_{i-1}\right) = 
\left(b_j - b_{j-1}\right)\]
\item[3. {\em CLASSIFY:}]~\newline
The ranges are then classified
into those seen in BEST\% {\em best} or {\em rest}.
\item[4.
{\em RANK:}]~\newline
The ranges are then ranked in increasing order either using Support-Based Bayesian Ranking \tion{sec:bayrank}  or Score-Based Energy Ranking \tion{sec:enrank}.
\item[5.
{\em PRUNE:}]~\newline
STAR2 then runs $K_2$ experiments with the models where the top ranked ranges
$1..X$ ranges are pre-set and the
remaining ranges can be selected at random.
In terms of the business case for automated software engineering tools,
this step is crucial since it would rank new ASE tools alongside 
other methods. 
\item[6.
{\em REPORT:}]~\newline
STAR2 returns the $1..X$ ranges that  produce the least effort, defects, and threats.
\end{description}

To run our experiments, we had to apply our engineering judgment to set the parameters. The following are the default values:
\[K_1=10,000, K_2=1,000, D=10, BEST=10\%\]

\newpage
\section{Results}
Our results are shown in \fig{exlarge}, \fig{exmedium} and
\fig{exsmall} for large, medium, and small systems (respectively).
Each dot in those figures represents a selected action to reduce
effort, threats, months, and defects.

Each table of these results divide up according to:
\bi
\item what can be controlled
such as the the strategic, tactical, or all factors: see s, t, a (respectively);
\item
whether or not prec increases or decreases (see p+, p- respectively)
\item
whether or not stor increases or decreases (see s+, s- respectively)
\item
whether or not cpu time increases or decreases (see t+, t- respectively)
\ei
The bottom rows of each table show the percent reduction in the
efforts, defects, threats, months of the system comparing the initial
unconstrained estimate and the estimate at the final minimum point.
For example, in \fig{exlarge}, the decisions in the first column (for 
{\em p-,s-,t-} and 
allowing control of 
``all'' variables) reduces effort to 28\% of the initial unconstrained estimate.
That is, the net benefit of this tool is a nearly four-fold reduction in the
predicted development effort.

\fig{exlarge}, \fig{exmedium} and
\fig{exsmall} are complex to read. 
\fig{commons} shows a summary of the number of times ranges
appear in the {\em strategic} runs of our results.
\begin{figure}[!h]
\begin{center}
\begin{tabular}{ccc|l}
small& medium &large\\\hline
8&8&8&acap=4,4.5\\
8&8&8&apex=4, 4.55.5\\
8&8&8&site=4.5\\
8&8&8&pmat=4.5,5\\
8&8&5&prec=4,5\\
8&8&6&pcon=2,3\\
1&2&3&peer=6\\
0&3&3&aa=5.5,6 \\
0&2&3&ett = 5.5,6
\end{tabular}
\end{center}
\caption[Frequent Ranges]{Number of times attribute ranges are mentioned in
\fig{exlarge}, \fig{exmedium} and
\fig{exsmall}.
Note that the maximum possible counts in this figure is eight.
That is, acap-4,4.5 (for example) 
is part of the best solution in $\frac{8}{8}=100\%$
of the solutions.}\label{fig:commons}
\end{figure}

Note that only a few ranges appear commonly in these results.
That is, despite all the variance in these models,
there exist a core set of best changes to make that improve
predicted defects,threats,efforts and development time:
\bi
\item acap
\item apex
\item site
\item pmat
\item prec
\item pcon
\ei
This is not to say that these other factors are unimportant,
only that compared to every other factor, there effects
were not as beneficial or stable across the space
of possibilities. 
\input{gone3}
\clearpage
\section{Conclusion}
The important conclusion is that there
exist stable conclusions about 
possible future best practices for process improvement,
despite wide spread variation internal to the USC models.

However, this work is incomplete. The ranges discussed
above defining strategic and controllable factors, and the 
validity of these above ranges to describe NASA systems,
is yet to be confirmed. The next stage of this work is to 
\bi
\item
Convene a panel of NASA experts to generate NASA-specific
ranges
\item
Run these simulations over those ranges.
\ei
%The panel meeting was conducted during ICSE 2008 on May 13. \fig{pan1} and \fig{pan2} are photos of this panel.
%It was composed of eight seasoned experts in the field of software engineering:
%\bi
%\item
%Barry Boehm
%\item
%Vic Basili
%\item
%Ray Madachy
%\item
%Thomas Ostrand
%\item
%Deiter Rombach
%\item
%Rick Selby
%\item
%Elaine Weyeuker
%\ei
%
%These experts formed the part of the panel from whom insights concerning the future trends in the software industry was derived. They discussed projections concerning all the COCOMO II/COQUALMO factors. Note that the time frame for these projections is within a decade. Other assumptions concerning the future projects include the following:
%\bi
%\item
%The projects/applications are {\em NASA-like} applications.
%\item
%They are {\em mission critical real-time} applications.
%\item
%They are {\em contractor built}.
%\item
%They have a {\em long life time} (between 5 and 10 years).
%\ei
%
%%\begin{figure}[!t]
%%\begin{center}
%%\includegraphics[scale=0.35]{workshop/workshop2.png}
%%\end{center}\caption{Picture of the panel}\label{fig:pan2}
%%\end{figure}
%%
%Three facilitators were present as well:
%\bi
%\item
%Bojan Cukic
%\item
%Tim Menzies
%\item
%David Raffo
%\ei
%
%The role of the facilitators was to guide the discussion on the model attribute projections in an attempt to keep the discussions focused. Gregory Gay acted as scribe during this session in order to transcribe notes from the meeting.
%
%During discussions, consensus was rarely reached concerning projections about how the software environment might change relative to the models. For this reason, after brief discussions about every attribute, a vote was taken concerning the future behavior of the attribute. These behaviors include the following:
%\bi
%\item
%\textbf{Bump}: The attribute would increase in value at first, and then decrease later on. PREC was voted to have such behavior.
%\item
%\textbf{Rick}: This is the opposite of bump, where attribute would decrease in value at first, and then increase later on.
%\item
%\textbf{Up}: The attribute would increase in value. ACAP was voted to have such behavior.
%\item
%\textbf{Down}: The attribute would decrease in value. APEX was voted to have such behavior.
%\ei
%
%Some chose to abstain from some of the votes as well. \fig{votes1} and \fig{votes2} contain the summary of the results of these discussions, as well as the votes.
%
%Note that in order to have future projections, we had to set a baseline that represent the current trends in software engineering. This had been set prior to the panel session with input from one of the expert panelists, Barry Boehm, and is presented in \fig{base}. From this baseline, we branched 8 main projections. the reason being that, as we see in \fig{votes1} and \fig{votes2}, only 3 attributes (PREC, STOR and TIME) were voted as not having a single direction of variation, namely bumps. \fig{cases} shows all the other attributes that were voted to have a single direction of change, and there corresponding ranges and values.
%
%
%	
%%\section{Other tools}
%\label {sec:other tools}
%
%In this paper, results from STAR2 are compared with results from two COCOMO II effort estimation tools. These tools are SCAT and 2CEE.
%
%\subsection {SCAT}
%SCAT is an acronym for Software Cost Analysis Tool.  It is an Excel-based version of the University of Southern California’s Constructive Cost Model II (COCOMO II). SCAT is dependent on a pre-calibrated COCOMO II model, as it is unable to perform local calibration. SCAT is also an adaptation to COCOMO II that allows the use of a range of inputs (Low, Most Likely, High) to capture uncertainty and generate probability distributions through the use of Monte Carlo techniques.  Therefore, unlike the USC version of COCOMO, which produces a single point estimate, SCAT produces distributional estimates in the form of cumulative distribution functions (also known as “S-curves”).
%
%The purpose of this tool is to provide software cost estimates using a Monte Carlo simulation in combination with COCOMO II cost drivers and algorithms. SCAT is the current tool used within JPL, and is therefore considered to be the baseline in our case studies.
%
%\subsection{2CEE}
%
%2CEE is a COCOMO II based software effort estimation tool. 2CEE is the brainchild of collaboration between NASA’s Jet Propulsion Laboratory (JPL) and West Virginia University (WVU). It is designed to explore the uncertainty in the model and in the estimate, to allow estimates early in the life cycle by representing new projects as ranges of values, and to provide numerous calibration options including stratification and feature selection.  By automating many tasks for the user it provides gains in cost analyst efficiency.
%
%2CEE uses historical project data for COCOMO II model calibration and estimation of COCOMO II effort for new projects. Furthermore, the estimates are generated through a Monte Carlo simulation, with the ability to run an n-fold cross validation on available data.
%


%\section{Recommendations for Future Work}
%
%Keep a COCOMO authority in the room (term definition problem).
%
%Panel needs more structure. Need to talk to a seed model.
%
%
\newpage
\bibliographystyle{IEEEtran}
\bibliography{refs}
\end{document}


%In the $21^{st}$ century, it is common place to
%find
%large complex
%software systems running on multiple platforms being
%being maintained and extended by international communities, interacting only via electronic means:
%e.g. ECLIPSE or  the continually evolving LINUX project(s).
%Decades of research
%has yielded tools and techniques that allow even a single individual
%to built complex software: e.g. Stallman \& EMACS
%or Torvalds \& LINUX (v1.0) 
%or
%Matsumoto \&  RUBY.  
%None of this would have been possible with decades of research into automating, simplifying, and improving
%common software engineering (SE) tasks. 
%
%Having addressed the {\em generation} problem 
%the focus of the automated  SE community
%has changed.
 Much current ASE research concerns
{\em automatic analysis} of source code or better
{\em execution-based testing tools},
These tools might, say,
verify formal properties or
search for the fewest regression tests that exercise  most of the system.
Some of these
tools are ready for industrial use; e.g, SPIN~\cite{holzmann97} or 
JPF~\cite{have00}, just to name
a few.   

%Recently, the authors were asked to make the business case for introducing some
%of these new ASE tools into large NASA projects. This case proved difficult to make,
%due to an {\em anti-automation bias} and the {\em local tuning problem}.

Previously, we had made a business case for introducing these new ASE tools into large
present day NASA projects. We were able to do so by using STAR1 to produce stable 
results recommending such tools despite not having local tuning data. This result
contradicted the {\em anti-automation bias} present in literature.

\begin{figure}[!b]
\scriptsize
\begin{center}
\begin{tabular}{l@{}|l@{~}r@{~}l}
%& & relative\\
id & features  & relative weight\\\hline
1&Personnel/team capability  & 3.53	&\rule{17mm}{2mm}\\
2&  Product complexity & 2.38&	\rule{12mm}{2mm}\\
3& Time constraint & 1.63&	\rule{8mm}{2mm}\\
4&    Required software reliability & 1.54&	\rule{8mm}{2mm}\\
5&    Multi-site development   &	1.53&	\rule{7mm}{2mm}\\
6&       Doc. match to life cycle  & 	1.52	&\rule{7mm}{2mm}\\
7&  Personnel continuity  &	1.51&	\rule{7mm}{2mm}\\
8&  Applications experience & 1.51	&\rule{7mm}{2mm}\\
9& Use of software tools & 1.50	&\rule{7mm}{2mm}\\
10& Platform volatility & 1.49&	\rule{7mm}{2mm}\\
11& Storage constraint & 1.46&	\rule{7mm}{2mm}\\
12& Process maturity  &	1.43	&\rule{7mm}{2mm}\\
13&    Language \& tools experience  &	1.43	&\rule{7mm}{2mm}\\
14&     Required dev. schedule & 1.43	&\rule{7mm}{2mm}\\
15&Data base size & 1.42&	\rule{7mm}{2mm}\\
16&  Platform experience & 1.40&	\rule{7mm}{2mm}\\
17&     Arch. \& risk resolution  &	1.39&	\rule{7mm}{2mm}\\
18&  Precedentedness  &	1.33&	\rule{6mm}{2mm}\\
19&  Developed for reuse  	&1.31	&\rule{6mm}{2mm}\\
20& Team cohesion  &	1.29&	\rule{6mm}{2mm}\\
21&    Development mode   &	1.32	&\rule{6mm}{2mm}\\
22& Development flexibility &	1.26	&\rule{6mm}{2mm}\\
\end{tabular}
\end{center}
\caption{Relative effects  on development effort.
Data from
a regression analysis of 161 projects~\protect\cite{boehm00a}.  
}\label{fig:coceffects}
\end{figure}

The {\em anti-automation bias} was seen
at an ICSE 2007 panel.
Tim Lister (a co-author of Peopleware~\cite{demarco87})
commented
that ``sociology beats technology
in terms of successfully completing projects''- a notion endorsed by the other panelists.
That is,
software project managers should focus {\em less}
on new ASE tools
and {\em
more} on managing the sociology aspects of their team (e.g. decrease
staff turnover).

\fig{coceffects} offers some support for this bias.
This figure shows the known relative productivity effects
of changing project features.
According to this figure,
the benefits of automatic tools (ranked number nine in the list) can be equaled or bettered
via other means (e.g. any item 1 to 8 {\em or}  any {\em pair} of items 
10 to 22).

Note that this support for the anti-automation bias is based solely
on the development effort; i.e.
\fig{coceffects}
is blind to the impact of new ASE tools in reducing {\em defects}
and any  other {\em threats} to the success of the project.
A complete business case should therefore 
study
predictors for effort {\em and} detects {\em and} threats using, for example:
\bi
\item
The COQUALMO defect predictor~\cite[p254-268]{boehm00b};
\item
The  COCOMO effort predictor~\cite[p29-57]{boehm00b}; 
\item
The THREAT  predictor for effort \& schedule overrun~\cite[284-291]{boehm00b}.
\item
The COCOMO schedule predictor for development time~\cite{boehm00b}.
\ei

Such a study is complicated by the {\em local tuning problem}.
These predictors are 
most accurate {\em after} they have been
tuned to local data (see \tion{sec:ev}). Unfortunately, 
   the data required for
   local tuning is difficult to obtain, especially across multiple
   organizations~\cite{me06d}.  This is due to  the business sensitivity associated
   with the data as well as differences in how the metrics are defined,
   collected and archived.  In many cases the required data has not been
   archived at all.  For example, after two years we were only able to
add 7 records to our NASA wide software cost metrics repository. In addition,
for the purposes of this paper, it is impossible to obtain such local tuning data
for the state of SE 10 years into the future.

The premise of this paper is that, in terms of ranking different methods
to be used in the future, a {\em precise} local tuning is not required.
Rather we need only to obtain general speculative data, from present day SE experts,
concerning future trends in the industry in order to provide guidance for the predictive models.
Based on research dating back to 1981~\cite{boehm81}, we assert that the
space of {\em
possible} tunings is well known. This space can be explored to
find features that minimize effort, defects, threats, and development time.  

To test that premise, we did the following:
\bi
\item
We built the STAR2 algorithm.
STAR2 is an evolved version of STAR1 which also uses a
simulated annealer
to sample the space of possible local tunings within
COCOMO, COQUALMO, THREAT, and SCHEDULE (development time). 
A standard SA
offers constraints to all variables. STAR2, on the other hand,
uses a Bayesian method to rank those constraints 
according to how
well they reduce
effort/ defects/ threats/ schedule.
Superfluous details can hence be deleted and the algorithm can return
a minimal set of project decisions 
that most reduce effort {\em and} defects {\em and} threats {\em and} schedule.
\item
At ICSE 2008, on May 13, a panel of SE experts was formed in order provide
insight into the future trends in the industry. While it proved hard
to get consensus concerning trends, and while the discussions concerning such trends
were aplenty, we were still able to obtain a good amount of data to characterize future
trends and form several projections concerning the future of SE. Our experiments and
analysis if based on these projections.
\ei

The general contributions of this paper, hence, are:
\bi
\item
%A {\em novel search engine} for sampling, then pruning a space of options within a model.
%\item
A {\em novel combination} of effort/defect/threat/schedule prediction models.
Previously, these three models have been analyzed one at a time or in pairs~\cite{me00e,me02f}.
\item 
A   {\em new solution} to the local tuning problem:
while
waiting for
local tuning data (which may never arrive),  
seek stable conclusions over the set of possible tunings.
\item
A look at the {\em future of Software trends} based on the panel discussions conducted with experts
from the field.
%\item
%A removal of the {\em tuning bias} problem. Predictors tuned to historical data can be biased
%by that data. This work, on the other hand, makes new conclusions far less susceptible to historical
%biases since it reports conclusions from across the space of biases. 
\item
A {\em demonstration} of our ability use present day SE expert speculation to assess
appropriate policies and actions to be used as the industry evolves in the medium to long term.
%that 
%the relative merits of different project decisions can be assessed
%{\em without} specific local tuning data.
\ei

\textbf{change this}
More specifically, using STAR2 we
we found situations where  
new ASE tools were {\em optional}  for
one NASA systems but {\em essential} for another 
(measured in terms of reducing effort {\em and} defects
{\em and} threats). Also, we did not find an {\em either/or} situation
where we had to choose, as suggested by Lister, between sociology
and technology.  When
ASE tools were useful, they were useful {\em in combination} with the sociological
features.


\textbf{The rest of this paper documents the COCOMO, COQUALMO, THREAT, and SCHEDULE models, along with the STAR2 algorithm and 2 other estimation tools : SCAT and 2CEE, and the Tactical and Strategic analysis. STAR2 is then applied on 8 variations of the predicted future trends of SE based on the results from the aforementioned panel. Each of these variations are tested as small, medium and large projects, which are in turn analyzed strategically, tactically, and fully using STAR2 for a grand total of 72 case studies. The method for obtaining the necessary data to generate the case studies will be presented. This is followed by a complete analysis of the case studies by STAR2. The paper is then concluded with notes on external validity and related work.}


\section{Definitions}\label{sec:inc}
Before we can
assess the relative merits of new ASE tools 
versus other methods, we must first define ``new ASE tools'' and ``other methods''.
Such a definition can be achieved via the ontology of COCOMO, COQUALMO, and THREAT, shown in \fig{cparems}.
This figure lists a variety of project {\em features} with the 
{\em range}
\{very low, low, nominal, high, very high, extremely high\} or
\[\{vl=1, l=2, n=3, h=4, vh=5, xh=6\}\]
Lister's sociological features occur many times
within \fig{cparems}. For example, {\em team} refers to the sociology
within a development team and its cohesiveness while {\em pcon}
refers to the staff turnover rate within a project.
 Also listed in \fig{cparems}
are factors like {\em acap, pcap} representing analyst and programmer
capabilities (respectively).

\input{cocomo}
As to technological features, new ASE tools
appear as
{\em execution-based testing tools} and
{\em automated analysis}.
Chulani~\cite{chulani99a} defines
the top half of 
{\em automated analysis} as:
\bi
\item[4] (high):
intermediate-level module and 
inter-module code syntax and 
semantic analysis. 
Simple requirements/design 
view consistency  checking. 
\item[5] (very high):
More elaborate 
requirements/design view 
consistency checking. 
Basic distributed-processing 
and temporal analysis, model 
checking, symbolic execution. 
\item[6] (extremely high):
Formalized\footnote{
Consistency-checkable pre- 
conditions and post-conditions, 
but not necessarily mathematical theorems.} 
specification and 
verification. 
Advanced distributed 
processing and temporal 
analysis, model checking, 
symbolic execution. 
\ei

\noindent
The upper half of {\em execution-based testing and tools} is:
\bi
\item[4] (high):
Well-defined test sequence 
tailored to organization 
(acceptance / alpha / beta / 
flight / etc.) test. 
Basic test coverage tools, test 
support system. 
\item[5] (very high):
More advanced test tools, test 
data preparation, basic test 
oracle support, distributed 
monitoring and analysis, 
assertion checking. 
Metrics-based test process 
management. 
\item[6] (extremely high):
Highly advanced tools for test 
oracles, distributed monitoring 
and analysis, assertion 
checking 
Integration of automated 
analysis and test tools. 
Model-based test process 
management. 
\ei

A  review of recent proceedings of the IEEE ASE conferences suggests
that 
a range of five or six in the above features includes
the kind of
``new ASE tools'' explored at this venue.
Hence, to compare ``new ASE tools'' to ``other methods'',
we will try to reduce effort and defects and threats
using  just
\[
\begin{array}{c}
\mbox{{\em automated analysis}}\in \{5,6\}\;\vee \\
\mbox{{\em execution-based testing and tools}} \in \{5,6\}
\end{array}
\]
or ``other methods''
(i.e. other ranges of \fig{cparems}).


%\subsection{Inherent Bias?}
%
%As mentioned above, our research funding depends on the continued interest
%of funding agencies in our work.
%It might hence  be argued that the authors are the {\em worst} team to comparatively
%evaluate automatic tools vs manual methods.
%It is therefore prudent to question our
%evaluation bias.
%
%To enable other researchers to question our biases,
%we take
%care to make our case studies fully reproducible.
%Reproducibility is an important methodological principle in other
%disciplines since it allows a community to confirm, refute, or even
%improve prior results.  In our view, in the field of software
%engineering, there are all too few examples of reproduced, and
%extended, results\footnote{ Exception: see  the reports of the
%PROMISE workshop \url{http://promisedata.org/repository/papers.html}}.
%Accordingly,
%an appendix to this paper detailed how to install and run the
%software required to repeat the KC1\&ARES case studies.
%Such reproducibility allows other researchers to check if our advocacy
%of automated tools biases our conclusions.
%
%\subsection{Is There Really a Problem?}
%
%To illustrate the local tuning problem,
%consider
%the COCOMO-81 development effort model~\cite{boehm81}.
%This model assumes 
%that the development $effort_i$ for $project_i$ is exponential on program size:
%\begin{eqnarray}\label{eq:ab} 
%effort_i &= &a \cdots KSLOC ^ b \cdots project_i\\
%project_i &= &\prod^{15}_j EM_j
%\end{eqnarray}
%Here, $project_i$ is computed from 
%the
%15 {\em effort multipliers} $EM_i$ shown in \fig{cparems}.
%{\em Local calibration} (LC)~\cite[p526-529]{boehm81} is a linear regression method 
%that returns the
%$a,b$ values that minimizes the difference between efforts predicted by \eq{ab} and the actual values seen in
%$project_{1..i}$.
%
%Based on historical data,
%the COCOMO-81 model offered three sets of cliched $a,b$ values that varied
%according to the different development {\em modes} of \fig{eso}.
%Subsequent work~\cite{boehm00a}
%in the late 1990s updated these values using 161 
%projects and,  even after that update, the $a,b$ values 
%were approximately equal to \fig{eso}.
%
%%\begin{figure}
%%\scriptsize
%%\begin{center}
%%  \begin{tabular}{rr|rr}
%%       a & b      &  a & b\\\hline
%%       3.72 &1.09 &  5.45 &1.05\\
%%       4.06 &1.08 &  7.92 &0.85\\
%%       4.22 &1.09 &  9.18 &0.85\\
%%\end{tabular}
%%\end{center}
%%\caption{Local tunings on $<a,b>$.}\label{fig:tune}
%%\end{figure}
%Reusing calibrations from other projects
%can lead to very inaccurate predictions.
%Off-the-shelf ``untuned'' models have 
%been up to 600\% inaccurate in their estimates, e.g. ~\cite[p165]{mukho92}~\cite{kemerer87}. 
%For example, \fig{lc} shows 
%local tuning experiments with some NASA COCOMO data\footnote{
%\url{http://promisedata.org/repository/data/nasa93}}.
%Local tuning was 
%performed on  ten sub-domains of the NASA data (flight, ground, avionics monitoring, etc).
%For each sub-domain, thirty times, 90\% of the data
%was randomly selected for local tuning via LC.
%The mean $<a,b>$ values are very different to the standard COCOMO-81 values of \fig{eso}.
%That is, specialized organizations like NASA develop software that is very
%different from the kinds of projects used to generate standard software process models.
%For such organizations, models can't be reused verbatim from some prior setting.
%
%Note the \hmm{underlined} entries
%of \fig{lc}. These entries show that  
%the standard deviation of the values learned from local tuning may be   very large.
%That is, in these NASA COCOMO data sets,
%there was insufficient data in these sub-domains to reach stable conclusions.
%That is,
%even if some local tuning data is available, it may still be incomplete.
%
%Rather than rely on missing or incomplete
%local data for model tuning, the rest of this paper explores how high-end
%ASE tools
%compare, relative to and manual methods, over the
%the space of possible tunings within the COCOMO, COQUALMO, and THREAT
% software process models.
%\begin{figure}[!t] 
%{\scriptsize 
%\begin{center}
%\begin{tabular}{p{0.5in}|r|r|p{1.8in}} \noindent Mode & a & b & notes\\\hline
%Organic & 3.2 & 1.05 & projects from relatively small software teams
%develop in-house software in a highly familiar environment.\\\hline
%Embedded & 2.8& 1.2 & projects operating within (is embedded in) a
%strongly coupled complex of hardware, software, regulations, and
%operational procedures.\\\hline Semi-Detached & 3.0 & 1.12 & An
%intermediary mode between organic and embedded.  
%\end{tabular}
%\end{center}} 
%\caption{COCOMO-81 development modes.}\label{fig:eso} 
%\end{figure}
%\begin{figure}[!t]
%\scriptsize
%\begin{center}
%\begin{tabular}{|rr|rr|}
%\multicolumn{2}{c}{a}
%&\multicolumn{2}{c}{b}\\\hline
%$\mu$&$\sigma$  & $\mu$ & $\sigma$ \\\hline
%3.72& 0.51& 1.09& 0.037\\
%4.06& 0.63& 1.07& 0.04\\
%4.27& 0.78& 1.02& 0.05\\
%5.06& \hmm{2.08} & 1.02& 0.09\\
%5.74 & \hmm{2.13}& 1.00 &0.084\\
%5.78& 1.5&0.85& 0.094\\
%6.28& 0.46& 0.93& 0.02\\
%6.35& 1.26& 0.97& 0.05\\
%6.35& 1.49& 0.92& 0.05\\
%9.18& \hmm{2.97}&0.88& 0.081\\\hline
%\end{tabular}
%\end{center}
%\caption{30 repeats of local tuning on  90\% samples. }\label{fig:lc}
%\end{figure}
%

\textbf{We will end this section with the definition of Tactical and Strategic analysis. Compared to full project analysis, both Tactical and Strategic analysis seek to determine recommendations for a subset of the COCOMO II parameters. Compared to each other, Tactical analysis is concerned with parameters that can be changed in the short term during the execution of the project, while Strategic analysis is concerned with parameters that are usually modified in the course of long term planning for the project.}

\textbf{The parameters that Tactical analysis is concerned with are: 
flex, 
resl, 
cplx, 
data, 
docu, 
ruse,
stor, 
tool,
sced.}

\textbf{The parameters that Strategic analysis is concerned with are: 
prec, 
pmat, 
acap, 
pcap, 
pcon, 
apex, 
plex, 
ltex, 
site.}

\textbf{All these parameters are briefly presented in \fig{cparems}.}

\input{models}

\section{STAR2}
Having defined the internal space of our models, we now search it.
A simulated annealer is used to  sample that space and a back select
algorithm is used to  rank feature ranges according to how well they reduce effort/ defects/ 
threats. This section will offer a brief explanation of these STAR2 internals.

\subsection{Simulated Annealing}
{\em Monte Carlo} algorithms randomly sample the space
of possible controllable model states. 
A  {\em Metropolis} Monte Carlo algorithm~\cite{metropolis53}
creates new states
by  small mutations to some
{\em current} state. 
If a new state is ``better'' (as assessed via an
{\em energy function}),  it becomes the new {\em current}
state used  for future
mutations.  Otherwise, a Boltzmann acceptance
criteria is used to probabilistically   decide to
assess the new state: the worse the new state, the less
likely that it becomes the new current state.
The algorithm is silent on the mutation mechanism. For our experiments,
we freeze $\frac{2}{3}$ of the features and
randomly  select ranges for the rest.

In 1983, Kirkpatrick et.el.~\cite{kirkpatrick83} 
proposed a modification that was
inspired
by a procedure used to make the strongest possible glass. Initially,
glass is heated to allow atoms to move freely. The temperature
is then slowly lowered such that the atoms can find the stablest
configuration with lowest energy. 
A {\em simulated annealing} (SA) algorithm adds
a ``temperature'' variable to the Boltzmann accept criteria
such that, at high temperatures, it is more likely that the
algorithm will jump to a  new worst current state.
This allows the algorithm to jump out of local minima while
sampling the space of options. As the temperature cools,
such jumps become less likely and the algorithm reverts
to a simple hill climber.

With SA, better solutions have lower ``energy''
so the fitness function $E$ is defined such that $E \ge 0$ and
{\em lower} values are
$better$. 
For our purposes, we model effort ($Ef$) as the distance to
the origin of the 4-D space with defects ($De$), threats ($Th$), and schedule ($Mo$):
\begin{equation}\small\label{eq:eng}
\begin{array}{c}
RD = relydefect^{(RELY-3)}\\\\
E=\frac{\left(\sqrt{(\overline{Ef}*\alpha)^2 + (\overline{De}*(\beta+RD))^2 + (\overline{Th}*\gamma)^2 + (\overline{Mo}*\delta)^2}\right)}{\sqrt{\alpha^2 + (\beta+RD)^2 + \gamma^2 + \delta^2}}\\
\end{array}
\end{equation}
Here, $\overline{x}$ is a normalized value 
$0 \le \frac{x - min(x)}{max(x) - min(x)} \le 1$. 
Hence, our energy ranges $0 \le E \le 1$ and
{\em lower} energies are {\em better}. $\alpha$, $\beta$, $\gamma$, and $\delta$ 
are the weighting factors for the different models. $relydefect$ is a constant
that defines a relationship between RELY and the defect model: as RELY increases,
the weighting of the defect model increases as well.

The acceptance criteria $P$ for a new state is defined using
the current temperature $T$, 
the energy of the current solution ($e$),
and the energy of the new mutation ($en$):
\begin{equation}\small
P(e,en,T) = e^{(e -en)/T}
\end{equation}
$T$ is defined to $decrease$ as the simulator loops through $k= 1\ldots kmax$ iterations.
We use
$T = e^{-100*k/kmax}$.

\begin{figure}[!t]
\scriptsize
\begin{verbatim}
function sa(kmax) 
  s := s0; e := E(s)           // Initial state, energy.
  sb := s; eb := e             // Initial "best" solution
  k := 0                       // Energy evaluation count.
  while k < kmax and e > emax  // Loop
    sn := neighbour(s)         //   Pick some neighbour.
    en := E(sn)                //   Compute its energy.
    if en < eb then            //   Is this a new best?
      sb := sn; eb := en       //     Yes, save it.
    if random() < P(e, en, temp(k/kmax)) 
     then s := sn; e := en     //     Maybe jump
    k := k + 1                 //   One more evaluation done
  return sb                    // Return best
\end{verbatim}
\caption{SA pseudo-code:
a new solution $sn$ (with new energy $en$) replaces the current solution                                  
if (a)~it has a lower energy or (b)~the acceptance predicate $P$
endorses it. Only in the case of (a)
should the new solution
replaces the current best solution.
} \label{fig:sa}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[width=2.5in]{jpl1.pdf}
%eps/JPL.pdf}
\end{center}
\caption{Processing a JPL requirements model. }\label{fig:jpl}
\end{figure}
Two advantages of SA are their {\em implementation simplicity} and their {\em ability to handle
non-linear models}:
\bi
\item {\em Implementation simplicity:}
\fig{sa} illustrates the simplicity of the algorithm. Memory is only required
for one current solution ($s$), one new solution ($sn$) and one best solution ($sb$) that stores
the best solution seen at any time in the simulation.
\item
{\em Non-linear models:}
 Previously~\cite{fea02a,me03i}, we have applied SA to non-linear JPL requirements models
where minimizing the cost of project mitigations can decrease the number of requirements
achieved by that project. Hence, decreasing {\em both} the cost and achieved requirements
is a non-linear problem that must trade between minimizing cost and increasing
requirements coverage.
The top-left line  of \fig{jpl} divides the behavior of the JPL requirements models
before  and after  simulated annealing. 
As shown below the line, 
initial Monte Carlo sampling of the possible mitigations lead to a large range
of costs and benefits. Simulated annealing found a set of mitigations
that lead to the small cloud of solutions above the line. Compared to the initial samples, these
new solutions had decreased cost, increased benefit (number of requirements covered), and decreased variance (shrank
the space of solutions). 
\ei
Two disadvantages of SA algorithms are their {\em incompleteness} and the {\em
complexity}
of their solutions:
\bi
\item 
{\em Incompleteness:} In our domain, we have some evidence that the incomplete nature of the heuristic SA
search is not a major problem.
\fig{saeg} shows a sample run of our SA tool running on our prediction models
for $K=10,000$ simulations. 
As $k$ increases for $1{\ldots}10,000$, it becomes less and less
likely that a better $best$ has been missed. Hence, we run our simulations
for ten times the period it takes for $best$ to stabilize (at $k\approx 1000$). 
\begin{figure}
\begin{center}
\includegraphics[width=2.7in]{saeg1.pdf}
\end{center}
\caption{Dots \& lines are
SA output from current \& best solution (respectively)  after $k$ simulations.
}\label{fig:saeg}
\end{figure}
\item
{\em Solution complexity:}
Simulated annealers offer constraints to all controllable features. 
Often this
is an over-constrained solution since, 
in many domains, a repeated empirical result is a {\em feature subset selection}
effect; i.e. models that constrain M variables perform just as well, or better,
than models that constrain N variables ($M\ll N$)~\cite{kohavi97,hall03}.
For example, Kohavi~\cite{kohavi97}
studied some machine learners to find  that using just 19\% of the available features
{\em increased} prediction accuracy by just 2.14\% (on average).
For another example,
when feature subset selection was applied to the JPL requirements model of \fig{jpl}, we found that
up to $\frac{2}{3}$-rds of the features can be left unconstrained,
without effecting the conclusions~\cite{fea02a}. 
\ei
%
%\begin{figure}[!t]
%\scriptsize
%\begin{center}
%\begin{tabular}{|l|r|r|r|p{0.95cm}|}\cline{2-4}
%\multicolumn{1}{c}{}     & \multicolumn{3}{|c|}{av.  number of features} \\\hline    
%dataset         &before &after &$\frac{after}{before}\%$ &accuracy change\\\hline    
%breast cancer &10  &2.9 &29\% &+0.14\%   \\\hline    
%cleve           &13  &2.6 &2\%  &+5.89\%   \\\hline 
%   crx             &15  &2.9 &19\% &+4.49\%   \\\hline  
%  DNA             &180 &11  &6\% &+3.63\% \\\hline    
%horse-colic     &22  &2.8 &13\% &+1.63\%   \\\hline
% Pima            &8   &1   &13\% &+0.79\%   \\\hline   
% sick-euthyroid &25  &4   &16\% &+0.38\%   \\\hline    
%soybean         &35  &12.7 &36\% &+0.15\%  \\\hline\hline    
%average         &38.5&4.99 &19\% &+2.14\% \\\hline
%\end{tabular}
%\end{center}
%\caption{Feature subset selection results from~\cite{kohavi97}}\label{fig:fssr}
%\end{figure}

In terms of  the goal of this paper, exploring feature subset selection
is very important.  Before we can {\em remove} non-essential features,
we must first  {\em rank} them according to their effectiveness.
As we shall see, this ranking will be insightful to the task of
assessing the relative value of new
 ASE
tools and other methods
for reducing effort/ defects/ threats.

\subsection{Ranking Methods}

\textbf{There are two ranking methods that can be used by STAR2. These are {\em Bayesian ranking} and {\em Energy ranking}. The former is the main ranking method used by STAR2, while the latter is included in STAR2 for verifying that the Bayesian ranking is valid.}

\subsubsection{Support-Based Bayesian Ranking}\label{sec:bayrank}

STAR2 ranks the features ranges
seen in $K$ runs of a simulated annealer by dividing the $K$ runs into:
\bi
\item {\em Best:} those associated with the  BEST\% solutions
(i.e. those with the BEST\% {\em least} energy); 
\item And the {\em rest} (i.e. the other 100-BEST\% of solutions).
\ei
It then computes the probability that a range is found in $best$.
using Bayes' Theorem.
Informally,
the
theorem says $next=old*new$ i.e. what we'll believe {\em next}
comes
from how {\em new} evidence effects {\em old} beliefs.  More
formally:\begin{equation}\label{eq:n}\small
P(H|E)=P(E|H) P(H)\; /\; P(E)\end{equation}
i.e. using
evidence $E$ and a prior probability $P(H)$ for
hypothesis $H\in\{best, rest\}$.
The theorem calculate a  posteriori  probability
$P(H|E)$. 
Simple  Bayes classifiers are often called ``n\aive'' since they assume
independence of each feature. While this assumption
 simplifies the implementation  
(frequency
counts are required only for each feature),
 it is possible
that
correlated events are missed by this ``n\aive'' approach.
Domingos and Pazzini show theoretically
that the independence assumption is a problem in a vanishingly small percent
of cases~\cite{domingos97optimality}. This explains the repeated
empirical result that, on average, seemingly n{\aive} Bayes classifiers perform as well
as other seemingly more sophisticated schemes (e.g. see Table~1 in~\cite{domingos97optimality}).


When applying the theorem, {\em likelihoods}
are computed from observed
frequencies, then normalize to create
probabilities (this normalization cancels out $P(E)$ in \eq{n}, so it need not be computed).
For example, after $K=10,000$ runs divided into  1,000 lowest 10\% {\em best} solutions and 9,000
{\em rest}, the  range $rely=vh$ might appears 10 times in the $best$ solutions, but only
5 times in the $rest$. Hence:

{\scriptsize
\begin{eqnarray}\nonumber
E&=& (reply=vh)\\\nonumber
P(best)&=&1000/10000 = 0.1\\\nonumber
P(rest)&=&9000/10000 = 0.9\\\nonumber
freq(E|best)&=&10/1000 = 0.01\\\nonumber
freq(E|rest)&=&5/9000 = 0.00056\\\nonumber
like(best|E)&=&freq(E|best) \cdot P(best)=0.001\\\nonumber
like(rest|E)&=&freq(E|rest) \cdot P(rest)= 0.000504\\\label{eq:ten}
P(best|E)&=& \frac{like(best|E)}{like(best|E) + like(rest|E)}=0.66
\end{eqnarray}}
Previously~\cite{clark05}
we have found that  \eq{ten} 
is a poor ranking heuristic
since it is distracted by low frequency evidence.
For example, note how
the probability of $E$ belonging to the best class
is moderately high even though its support is very low; i.e. $P(best|E)=0.66$ but
$freq(E|best) = 0.01$. 

To avoid such unreliable low frequency evidence, we augment \eq{ten}
with a support term.
Support should $increase$ as the frequency of a range $increases$,  i.e. 
$like(x|best)$ is a valid support measure.
STAR2 hence ranks ranges via
\begin{equation}\label{eq:ps}\small
P(best|E) * support(best|E) = \frac{like(x|best)^2}{like(x|best) + like(x|rest)}
\end{equation}

\subsubsection{Score-Based Energy Ranking}\label{sec:enrank}

While the main ranking algorithm used by STAR2 is the Bayesian method 
explained above, another very basic and intuitive baseline method was 
implemented to test Bayesian ranking. This method is Score-Based Energy
Ranking. Basically this ranking scheme uses the means of the results 
from the Simulated Annealing simulations to assess and rank COCOMO II 
attribute-value combinations. The mean results are based the scores 
generated by the Energy Function of the SA algorithm. These scores are 
basically the distance from the origin to a point in three dimensional 
space whose coordinates are the normalized Effort, Defects, and Threat scores from our models. Energy Ranking works as follows:

\bi
\item[1.] {\em DATA COLLECTION:}
This is done while the SA algorithm is 
running whenever the algorithm generates a new candidate best result.
 The attribute-value combinations that are used for that candidate are 
detected, and data concerning the score is stored for each of these used attribute-value combinations.
\newline
\item[2.] {\em MEAN SCORE CALCULATION:}
After the SA goes through its K 
runs, the mean scores for all the attribute-value combinations are calculated. 
After this step, the attribute-value combinations are ready to be ranked.
\ei


While Energy Ranking seems to be simple enough, is has one main 
potential weakness: outliers in simulations could taint the score of
an otherwise good attribute-value combination, causing its rank to 
decrease compared to seemingly worse combinations. This issue is not
present in Bayesian Ranking since it is dependent on frequency and support rather than scores.

\subsection{The STAR2 Algorithm}
To apply \eq{ps}, STAR2 runs in six phases.
In terms of standard machine 
learning theory, step 1 generates a training set; steps 2,3,4 do some generalization;
and step 5 tests the learned theory on data not seen during training.

\begin{description}
\item[1. {\em SAMPLE:}]
To sample the ranges from the models,
STAR2 runs the simulated annealer  $K_1$ times.
\item[2. {\em DISCRETIZE:}]
The data seen in the $K_1$ samples is then discretized into $D=10$ bins.
Discretization converts a continuous range into a histogram
with $n$ break points $b_1\ldots b_n$ where $\left(\forall i < j : b_i \le b_j\right)$. 
After discretization,
many observations can fall into the same range
between $b_i$ and $b_{i+1}$
at frequency counts $c_i$. This study used equal width discretization; i.e. 
\[\forall i,j:
\left(b_i - b_{i-1}\right) = 
\left(b_j - b_{j-1}\right)\]
\item[3. {\em CLASSIFY:}]
The ranges are then classified
into those seen in BEST\% {\em best} or {\em rest}.
\item[4.
{\em RANK:}]
The ranges are then ranked in increasing order either using Support-Based Bayesian Ranking \tion{sec:bayrank}  or Score-Based Energy Ranking \tion{sec:enrank}.
\item[5.
{\em PRUNE:}]
STAR2 then runs $K_2$ experiments with the models where the top ranked ranges
$1..X$ ranges are pre-set and the
remaining ranges can be selected at random.
In terms of the business case for automated software engineering tools,
this step is crucial since it would rank new ASE tools alongside 
other methods. 
\item[6.
{\em REPORT:}]
STAR2 returns the $1..X$ ranges that  produce the least effort, defects, and threats.
\end{description}

To run our experiments, we had to apply our engineering judgment to set the parameters. The following are the default values:
\[K_1=10,000, K_2=1,000, D=10, BEST=10\%\]

%Initially, we planned experiments that varied these parameters,
%as well as trying other discretization policies.
%However, our initial results were so promising (see below) that we are not currently motivated
%to explore 
%other parameter settings (such an exploration is planned for future work).

Note that for Energy Ranking, the algorithm is slightly different. Energy ranking does not require the disretizing and the classifying steps 


%\begin{figure}[!t]
%\begin{center}
%\scriptsize
%\begin{tabular}{c|lrr|lr}
%       &\multicolumn{3}{c|}{ranges}      &\multicolumn{2}{c}{values}\\\hline
%project&feature&low&high&feature&setting\\\hline
%&rely&3&5&tool&2\\
%Flight:&data&2&3&sced&3\\
%&cplx&3&6&&\\
%&time&3&4&&\\
%&stor&3&4&&\\
%&pvol&2&4&&\\
%&acap&3&5&&\\
%&apex&2&5&&\\
%&pcap&3&5&&\\
%&plex&1&4&&\\
%&ltex&1&4&&\\
%&pmat&2&3&&\\
%&Ksloc&7&418\\\hline
%&rely&1&4&tool&2\\
%Ground:&data&2&3&sced&3\\
%&cplx&1&4&&\\
%&time&3&4&&\\
%&stor&3&4&&\\
%&pvol&2&4&&\\
%&acap&3&5&&\\
%&apex&3&5&&\\
%&pcap&3&5&&\\
%&plex&1&4&&\\
%&ltex&1&4&&\\
%&pmat&2&3&&\\
%&Ksloc&11&392\\\hline
%&prec&1&2&data&3\\
%OSP:&flex&2&5&pvol&2\\
%Orbital&resl&1&3&rely&5\\
%space&team&2&3&pcap&3\\
%plane&pmat&1&4&plex&3\\
%&stor&3&5&site&3\\
%&ruse&2&4&&\\
%&docu&2&4&&\\
%&acap&2&3&&\\
%&pcon&2&3&&\\
%&apex&2&3&&\\
%&ltex&2&4&&\\
%&tool&2&3&&\\
%&sced&1&3&&\\
%&cplx&5&6\\
%&KSLOC&75&125\\\hline
%&prec&3&5&flex&3\\
%OSP2 &pmat&4&5&resl&4\\
%&docu&3&4&team&3\\
%&ltex&2&5&time&3\\
%&sced&2&4&stor&3\\
%&KSLOC&75&125&data&4\\
%&&&&pvol&3\\
%&&&&ruse&4\\
%&&&&rely&5\\
%&&&&acap&4\\
%&&&&pcap&3\\
%&&&&pcon&3\\
%&&&&apex&4\\
%&&&&plex&4\\
%&&&&tool&5\\
%&&&&cplx&4\\
%&&&&site&6\\
%\end{tabular}
%\end{center}
%\caption{Four of the five case studies.}\label{fig:cases}
%\end{figure}

\section{Data Collection}
In this section we will describe how data was collected in order to provide some insight on future trends in the software industry. That will be followed with presenting the results of this data collection and how those were used in STAR2.

\begin{figure}[!t]
\begin{center}
\includegraphics[scale=0.35]{workshop/workshop1.png}
\end{center}\caption{Picture of the panel}\label{fig:pan1}
\end{figure}

\input{icseMay13}

\begin{figure}[!t]
\begin{center}
\scriptsize
\begin{tabular}{l|c|l|c}
feature&setting&feature&setting\\\hline
aa&3&tool&3\\
acap&3&time&3\\
apex&4.5&team&3.5\\
cplx&3&stor&3\\
data&4&site&4.5\\
docu&3&sced&3\\
ett&2&ruse&4\\
flex&3&resl&3\\
ltex&4.5&rely&2.5\\
pcap&3&pvol&2\\
pcon&3&prec&4.5\\
peer&2.5&pmat&2.5\\
plex&4.5\\
\end{tabular}
\end{center}
\caption{The baseline settings representing present day software engineering}\label{fig:base}
\end{figure}

\begin{figure}[!t]
\begin{center}
\scriptsize
\begin{tabular}{lrr|lr}
       &\multicolumn{1}{c}{ranges}      &\multicolumn{3}{c}{values}\\\hline
feature&low&high&feature&setting\\\hline
aa&3&6&data&4\\
acap&3&5&docu&3\\
apex&1&4.5&ltex&4.5\\
cplx&3&6&pcap&3\\
ett&2&6&plex&4.5\\
flex&3&5&team&3.5\\
pcon&1&3&&\\
peer&2.5&6&&\\
pmat&2.5&5&&\\
pvol&2&5&&\\
rely&2.5&5&&\\
resl&3&5&&\\
ruse&2&6&&\\
sced&1&3&&\\
site&1&4.5&&\\
tool&3&5&&\\

\end{tabular}
\end{center}
\caption{Common settings for all future trend variants}\label{fig:cases}
\end{figure}

\input{gone3}

The panel meeting was conducted during ICSE 2008 on May 13. \fig{pan1} and \fig{pan2} are photos of this panel.
It was composed of eight seasoned experts in the field of software engineering:
\bi
\item
Barry Boehm
\item
Vic Basili
\item
Ray Madachy
\item
Thomas Ostrand
\item
Deiter Rombach
\item
Rick Selby
\item
Elaine Weyeuker
\ei

These experts formed the part of the panel from whom insights concerning the future trends in the software industry was derived. They discussed projections concerning all the COCOMO II/COQUALMO factors. Note that the time frame for these projections is within a decade. Other assumptions concerning the future projects include the following:
\bi
\item
The projects/applications are {\em NASA-like} applications.
\item
They are {\em mission critical real-time} applications.
\item
They are {\em contractor built}.
\item
They have a {\em long life time} (between 5 and 10 years).
\ei

\begin{figure}[!t]
\begin{center}
\includegraphics[scale=0.35]{workshop/workshop2.png}
\end{center}\caption{Picture of the panel}\label{fig:pan2}
\end{figure}

Three facilitators were present as well:
\bi
\item
Bojan Cukic
\item
Tim Menzies
\item
David Raffo
\ei

The role of the facilitators was to guide the discussion on the model attribute projections in an attempt to keep the discussions focused. Gregory Gay acted as scribe during this session in order to transcribe notes from the meeting.

During discussions, consensus was rarely reached concerning projections about how the software environment might change relative to the models. For this reason, after brief discussions about every attribute, a vote was taken concerning the future behavior of the attribute. These behaviors include the following:
\bi
\item
\textbf{Bump}: The attribute would increase in value at first, and then decrease later on. PREC was voted to have such behavior.
\item
\textbf{Rick}: This is the opposite of bump, where attribute would decrease in value at first, and then increase later on.
\item
\textbf{Up}: The attribute would increase in value. ACAP was voted to have such behavior.
\item
\textbf{Down}: The attribute would decrease in value. APEX was voted to have such behavior.
\ei

Some chose to abstain from some of the votes as well. \fig{votes1} and \fig{votes2} contain the summary of the results of these discussions, as well as the votes.

Note that in order to have future projections, we had to set a baseline that represent the current trends in software engineering. This had been set prior to the panel session with input from one of the expert panelists, Barry Boehm, and is presented in \fig{base}. From this baseline, we branched 8 main projections. the reason being that, as we see in \fig{votes1} and \fig{votes2}, only 3 attributes (PREC, STOR and TIME) were voted as not having a single direction of variation, namely bumps. \fig{cases} shows all the other attributes that were voted to have a single direction of change, and there corresponding ranges and values.


	
%\section{Other tools}
%\label {sec:other tools}
%
%In this paper, results from STAR2 are compared with results from two COCOMO II effort estimation tools. These tools are SCAT and 2CEE.
%
%\subsection {SCAT}
%SCAT is an acronym for Software Cost Analysis Tool.  It is an Excel-based version of the University of Southern California’s Constructive Cost Model II (COCOMO II). SCAT is dependent on a preclibrated COCOMO II model, as it is unable to perform local calibration. SCAT is also an adaptation to COCOMO II that allows the use of a range of inputs (Low, Most Likely, High) to capture uncertainty and generate probability distributions through the use of Monte Carlo techniques.  Therefore, unlike the USC version of COCOMO, which produces a single point estimate, SCAT produces distributional estimates in the form of cumulative distribution functions (also known as “S-curves”).
%
%The purpose of this tool is to provide software cost estimates using a Monte Carlo simulation in combination with COCOMO II cost drivers and algorithms. SCAT is the current tool used within JPL, and is therefore considered to be the baseline in our case studies.
%
%\subsection{2CEE}
%
%2CEE is a COCOMO II based software effort estimation tool. 2CEE is the brainchild of collaboration between NASA’s Jet Propulsion Laboratory (JPL) and West Virginia University (WVU). It is designed to explore the uncertainty in the model and in the estimate, to allow estimates early in the lifecycle by representing new projects as ranges of values, and to provide numerous calibration options including stratification and feature selection.  By automating many tasks for the user it provides gains in cost analyst efficiency.
%
%2CEE uses historical project data for COCOMO II model calibration and estimation of COCOMO II effort for new projects. Furthermore, the estimates are generated through a Monte Carlo simulation, with the ability to run an n-fold cross validation on available data.
%


\section{Experiments}

In this section, we will present experimental data related to STAR2. We will present how well STAR2 works in our case studies in terms of reducing project projections for effort, defects, threats, and development time. We will also analyse the recommended procedures and policies that STAR2 will recommend for the case studies which represent our future projections, and through those results show that ASE tools are not only important for future projects, but also that these tools need to be applied in an advanced manner.

As mentioned at the end of section \ref{sec:intro}, we have 72 individual case studies. The results of running these case studies are presented in \fig{exlarge}, \fig{exmedium} and \fig{exsmall}. Each of these figures are divided into 2 parts: the policies section and the \% reduction section. Note that $p$, $s$, and $t$ at the top of the tables correspond to PREC, STOR, and TIME. These are the attributes that vary between positive and negative gradients, as voted on in \fig{votes2}.

\subsection{policies}
When referring to \fig{exlarge}, \fig{exmedium} and \fig{exsmall}, we notice that the policies that are chosen by STAR2 are very similar across project sizes. Exceptions to this exist, such as the tendency of STAR2 to prefer larger values for SCED for larger projects when compared to the results of medium and small projects. This suggests that, for larger projects, the cost of speeding up a project schedule will  probably out way the benefits, as this might increase the amount of defects that end up in that large project, which would cost the project additional effort later on in the project life cycle. Keep in mind that one of our assumptions for the projects is that they possess a longer lifetime, which would emphsize such a project disadvantage.

Another effect that we observe is that there isn't as much variation between the policies of separate projections as we would expect. This would seem to suggest that, even though we have 8 separate base projections, the overall projection is stable and determinant enough to actually draw stable conclusions from that would apply across the board. In other words, for the purposes of STAR2, the experts panel from whom the projections originate don't necessarily need to have consensus for us to draw stable future predictions about policies that need to be applied in the future.

The effect that most interests us for the purposes of this article is related to ASE tools. On the one hand these tools are recommended for the majority of our case studies. This applies most to those cases with full or technical analysis, and to a lesser extent to those with strategic analysis. Note however that in all cases, whenever these tools are applied, they are applied at very high or extremely high levels 
(between 5 and 6), which emphasizes the importance of advanced ASE tools for future projects. These tools however have the most utility whenever the project is in the development phases. This explains our result since statistical analysis is more related to the planning stages of a project, which tactical analysis is more related to the development stage of a project. Also notice that for the same analysis methods the emphasise the use of advanced ASE tools, the TOOL is recommended at its highest value, which means having it integrated in to the life cycle according to \fig{cparams}. So there seems to be a correlation between having the need for ASE tools, and requiring having these Tools integrated into the software life cycle.

\subsection{Reductions}
The lower parts of \fig{exlarge}, \fig{exmedium} and \fig{exsmall} refer to the size of the final model prediction values compared to the initial ones, i.e. the ones before applying any kind of policies on the case study in question. So smaller is better. We notice four main effects in these tables.

The first effect is that the reductions are very similar across case studies of the same kind of analysis. This doesn't apply in all cases however. The exceptions include such cases were ASE tools aren't applied at all. These tend to have much lower reductions in defects compared to other cases with these tools were being applied. However, the overall effect seems to be that overall the reductions are similar for the same analysis types regardless of the projection or project size. This is mainly due to the similarity in the policies that are followed, which is mentioned in the previous section.

Secondly, effort reductions seem to be larger for strategic compared to tactical analysis. This would suggest that strategic long term planning is more effective at reducing cost for future projects. Note that our strategic analysis with STAR2 didn't decisively recommend the use of advanced ASE tools. So there seems to be an argument that ASE tools are necessary for cost reduction. However this isn't the whole picture.

We also observe that defect reductions are largest with tactical analysis. This is due to the affinity of this kind od analysis to recommend ASE tools be used. ASE tools are mainly defect reduction tools, so its only natural that the analysis that most recommends these tools achieves higher defect reductions. The fact that its tactical analysis suggests that ASE tools are best used during the development stages of a software product rather than during the planning stages.

The last effect that we observe is that full analysis achieves the best reductions by a large margin. This applies for all the models and all the case studies, suggesting that if we could control everything in a project, we should. This unfortunately isn't really possible nor feasible. However this can be interpreted as using both proper strategic and tactical planning when appropriate. So, use strategic planning on a future project, apply the policies, then transition to tactical planning and apply the associated policies. By doing this, a software project could achieve a similar effect to controlling all, or most, of the variables in a software project achieving high reductions in effort, defects, and months (schedule). Note that this type of analysis recommends used advanced ASE tools most of the time, providing a strong business case for them.

Note that the threat model doesn't seem to have much importance in our case studies in that model never seem to register any tangible values in any of the experiments. This might be because of some bug in the implementation of the model itself or in our reporting methods and scripts. However, it is more likely to be caused by the nature of the case studies themselves, since they aren't very constrained case studies. We have seen this before, where for less constrained case studies threat doesn't really come into play.

%STAR2 was tested on the four case studies of
%\fig{cases}:
%\bi
%\item OSP: The Orbital Space Plane GNC prototype (a 1990s NASA system).
%OSP was an early prototype for OSP2.
%\item OSP2: the guidance and navigation control system of a current NASA launch vehicle, under development.
%\item FLIGHT/GROUND: these two are project templates used for nasa projects.
%\ei
%In \fig{cases},  {\em values} are {\em fixed} while {\em ranges} represent a space of options.
%In the sequel, this observation will be important:
%OSP2
%constrains most of its features to fixed values while
%OSP allows more variation in feature ranges. Flight and Ground are more like project templates that, for the most part, enforce restrictions on the ranges of the project. OSP and OSP2 and both considered "Flight" projects.

%\subsection{STAR2 vs. SCAT vs. 2CEE}
%
%While STAR2 has been shown above to be effective as a reccomendation tool for reducing COCOMO effort, defects, and threat, we would like to compare it to other tools that are used for the purpose of reducing COCOMO effort. These tools are SCAT and 2CEE. Note that this comparison isn't exactly an apples to apples comparison, reason being that STAR2 takes a fundamentally different approach compared to those two other tools. While SCAT and 2CEE depend on having a calibrated COCOMO model, STAR2 bypasses that dependency by exploring the space of possible calibrations of the model, and attempting to reduce the COCOMO effort by presenting model recommendations to be applied by the project. It also differs in that it doesn't only work with simply one model, but rather works with a family of models, which in this case is the COCOMO family of models consisting of the Effort, Defects, and Risk models presented in \tion{}. 
%
%For this comparison we use the effort results from STAR2 associated with full analysis, along with results from the other two tools after running them on th same projects that STAR2 was run on.


%
% 
% SA was run $K_1$ times and
% ranges not mentioned in \fig{cases} selected at random from 
% Equations 2,4,5,7,8 and \tion{threat}.
% The selected {\em controllable} ranges were ranked with
% \eq{ps}:
% \bi
% \item
%  By {\em controllable} features, we mean the kinds of changes
% a project manager could make to her project; i.e. \fig{cparems}.
% \item
% {\em Uncontrollable} features
% are features like the correlation between, say, analyst capability and development effort;
% i.e.
% the internal model parameters of 
% Equations 2,4,5,7,8 and \tion{threat}.
% \ei
% \input{ospresults}
% \input{aresresults}
% 
% The list at the bottom of \fig{osp}
% shows how \eq{ps}  ranked
% OSP project decisions.
% These ranks correspond to the X-axis of the plots at the top of that figure.
% ``Median'' plots the 50\% percentile of the defect/ effort/ threat
% values seen
% after imposing ranges $1..X$ (and selecting all other ranges at random).
% ``Spread'' shows the 75\%-50\% percentile range.
% These plots are U-shaped:
% \bi
% \item On the left-hand-side of each plot,
% poor results were seen
% after applying too few constraints.
% That is, models perform poorly if we do not control them enough.
% \item
% On the right-hand-side of each plot, poor results were also seen.
% In terms of \eq{ps}, these are the ranges with low support and
% low probability of belonging to $best$ class. Hence, it is not surprising that
% applying these superfluous constraints was counter-productive.
% \ei
% 
% The ranges from new ASE tools (see the ``$\blacktriangle$'' symbol)
% are ranked at X=31, 37, 40, 41.
% These rankings can assessed using the minimum points
% in the median plots:
% \bi
% \item
% At $X\approx15$
% effort and threat are at their global minimums  while
% defects is at a local minimum.
% \item
% The global minimum for defects can be see at $X=33$. 
% At this point, defects are 30\% less than at $X=15$.
% However, this lower defect level is achieved via
% doubling effort
% from 1000 to 2000. months.
% \ei
% 
% \fig{osp} shows that OSP needs some 
% tool support.
% The third
% highest ranked range (X=3) is for $tool=3$; i.e.
% medium range tool use,
% This corresponds to some support
% for
% edit, code, and debug
% moderately integrated with processes and methods and reuse. 
% 
% However, \fig{osp} makes only a weak business case
% for more extensive tool support.
% {\em Automated analysis} 
% is ranked at positions 40,41 and
% is not required to reach
% the
% minimums of the  \fig{osp} plots.  On the other hand,
%  if management has the budget
% to support the X=33 effort, then using
% extremely high range {\em execution-based testing  and tools} is 
% strongly recommended. 
% 
% Clearly,
% if the project is cost-adverse, then methods other than
% new ASE tools are
% recommended. However, the OSP 
% business users should be aware of the 
% consequences of this policy.
% Applying only decisions $X=1..15$
% will produce
% a product costing half of the previous recommendation,
% but with 30\% more defects.
% 
% 
% While new ASE tools are {\em optional} for OSP, they are 
% {\em essential} for OSP2.
% \fig{ares}
% shows the OSP2 results.
% Minimum defects were achieved
% at X=17 via a {\em combination} of:
% \bi
% \item {\em technical features}:
% project decisions from the range
% $1 \le X \le 17$
% use
% three of our four new ASE tools (see X=11,14,15).
% \item and {\em sociological features}  such as $docu$,
% use
% of documentation; $pmat$, process maturity; $ltex$,
% language \& tool experience; and
% controlling $sced$ schedule pressure and the $prec$
% precedentedness  of the project.
% \ei
% Hence, the argument quoted in the  introduction that
% ``sociology beats technology''
% should  be corrected to ``sometimes,
% sociology can use more technology''.
% For example, in \fig{ares}, only sociological features
% are applied for $1\le X \le 10$. Observe how, at X=11,
% the largest drop in defects was achieved {\em after} applying
% extremely high range automated analysis.
% 
% 
% Two other noteworthy features 
% of \fig{ares} are:
% \bi
% \item
% The U-shapes seen in all the OSP results of \fig{osp} 
% only appears in OSP2's defect plot. 
% The median effort and threat curves for OSP2 are essentially flat. 
% As seen in \fig{cases}, OSP2 constrains more
% features than OSP. 
% Hence, at least for OSP2 effort and threats, STAR2 could not change
% those
% median values. 
% \item
% On the other hand,  
% the flatness of the effort and threats plots
% means that they are stable across a wide range of project settings and internal variations to
% the prediction models. For example, according to this study, the effort for OSP2 
% has 50\% to 75\% range of 600 to 900 person months (median
% plus spread). Such stability is very useful for project planning.
% \ei
\section{External Validity}\label{sec:ev}


One benefits of STAR2's   analysis is a
reduction in {\em sampling bias}.
Conclusions reached from prediction models tuned to local data are biased
by that tuning. The conclusions reached here, on the other hand, are stable
over a large space of possible tuning biases. This hence enables STAR2 
to analyse future projects and trends in the medium to long term 
without being limited by the lack of local tuning data for them.


Another issue is {\em model bias}.
The above conclusions are drawn from the internal space
of some software prediction models. Clearly, if
those predictions models are wrong,
then these conclusions are also wrong,
For example,
the reader might believe that this analysis has overlooked
the impact of features {\em not} mentioned in \fig{cparems}.

One mitigation for this risk is to
use the best available models.
The models used here
have been extensively validated:
\bi
\item
Chulani et.al.~\cite{chulani99} reports a one study with a Bayesian
tuning algorithm using 161 projects. After tuning, a cross-validation
study showed that COCOMO produced effort estimates that are 
within 30\% of the actuals,
69\% of the time.
\item
Studies with the COCOMO-81 project database have shown that
the THREAT index correlates with well
with the $\frac{months}{KDSI}$ (KDSI= thousand
of delivered source lines of code).
This result is consistent with the
the base premise of THREAT; i.e.  bad management can delay
software delivery.
\item
COQUALMO was developed using extensive feedback
from the  COCOMO affiliates
group. This group comprises 
dozens
of companies, that have donated 161 data sets, that meet each year to discuss
improvements to the current set of
models\footnote{\url{http://sunset.usc.edu/events/2006/CIIForum/}}.
\ei

Another threat to external
validity are the biases of the authors.
Given the intended publication venue,
and the authors' history of
publishing on
automatic SE tools,
perhaps this study is inherently biased
in favor of new ASE tools.

To enable other researchers
to question our biases, we take care to make our case studies fully
reproducible.  Reproducibility is an important methodological
principle in other disciplines since it allows a community to
confirm, refute, or even improve prior results.  In our view, in
the field of software engineering, there are all too few examples
of reproduced, and extended, results.
Accordingly,
an appendix to this paper details how to install and run the
software required to repeat the OSP\&OSP2 case studies.  Such
reproducibility allows other researchers to check if our advocacy
of automated tools biases our conclusions.

Yet another threat to external validity is our {\em evaluation bias}.
Tacit in \eq{eng} is an assumption that reductions in threat, effort,
and defects are of {\em equal} value. This may not be the case, especially
for safety critical software where 
reducing defects has top priority, regardless of the cost. 

Having documented
this problem, we now ignore it. This paper has shown that there exists at least
one evaluation bias under which ASE tools are essential to at least one project. For this
paper, that will suffice.

The final threat to external  validity is STAR2's 
{\em search bias}.
There are $2^N$
possible combinations
of $N$ ranges
but STAR2's PRUNE operator
 only explores $N$ of them.
Hence, in theory, the rankings of \fig{osp}
and \fig{ares}  may not accurately reflect 
the true relative effectiveness of the ranges.

There is some theoretical and empirical evidence that 
this search bias is not a major issue.
Clark has conducted a  theoretical study concluding that
\eq{ps} 
offers the wrong rankings in a very small 
percent of cases~\cite{clark05}.
To check his theory,
Clark ran numerous experiments comparing
(a)~the estimated ``best'' combinations
of ranges
found by \eq{ps} and the (b)~actual
best combinations found by generating 
combinations,  then trying each one against the 
training data.  In all those experiments,
Clark only found one minor case
where $ranking(a) \not= ranking(b)$.

Clark's results explains the U-shape seen in the above plots.
If the rankings of \eq{ps} were accurate
then, initially, 
performance should improve as an increasing number
of effective ranges are applied.
Eventually we run out of the effective ranges
and start applying ranges
with low probability and support of belonging
to the $best$ class. After that point, performance should
degrade. 
In the middle of these two extremes would be a low energy valley where
the most effective constraints have been applied.
Such   U-shape performance plots were seen in \fig{osp} and \fig{ares}.

Clark's analysis is too long to present here.
However, a small simulation illustrates his style of argument.
STAR2's PRUNE operator
ranks $N$ ranges in the order $1,2,3..N$. 
Consider a {\em greedy search} 
that returns the first $1,2,3...X\le N$ of these ranges.
Such a search grows $X$ till
some superset $1..Y$ performs worse the  subset $1..X$
($X < Y$).
For this greedy search to be optimal,
then 
 longer combinations $1..Y$
can't be more effective than
shorter combinations $1..X$. 

This condition can be checked via
simulation.
Consider the case of $N=100$ ranges classified into 1000 $best$ cases
and $9000$ $rest$ cases; i.e. $P(best)=10\%$,
and $P(rest)=90\%$.
A simulator
can randomly generates N=100 pairs $\{a,b\}$  
for $F(a|best)$ and $F(a|rest)$ (respectively)
where
\[
 1 \le a \le 1000 \wedge 1 \le b \le 9000\]
After \eq{ps} ranks the pairs, the simulator picks two combinations 
$1..X$ and $1..Y$ where $1\le X < Y \le100$.
According to \eq{ten},
98\% of time,
the shorter combination has a higher probability
than the longer one of being {\em best}.
That is,  STAR2's  linear search over \eq{ps}'s rankings
 1 to $N$
will not miss
interesting combinations. 

\section{Related Work}
This report is somewhat at odds with a standard IEEE ASE paper.
For example, there many
reports at this conference
of innovative tools that find defects missed by other
methods. Such reports offer strong support for the utility
of new ASE tools. However, note that they assess the new ASE
tools {\em soley} in terms of defect detection. In the paper, on
the other hand, we have tried to assess new ASE tools in the terms
of multiple success criteria (defect removal, effort, threats).

This paper was motivated by a lack of results regarding the relative
merits of new ASE tools versus different methods.  To be fair, apart from \fig{coceffects},
the general software engineering literature is weak on the relative
merits of {\em any} set of methods.  We recently reviewed the
(approx) 100 V\&V methods offered by the SE literature\footnote{
Wallace and Fujii's definitions of V\&V~\cite{wallace89};
the IEEE 2004 standard on V\&V~\cite{1012}; NASA's recommended V\&V
practices~\cite{jackson97}}. While some evidence was offered for
the value of general techniques\footnote{ E.g. improving software
maturity decreases rework~\cite{king02}} there was very little
precise evidence on the merits of a particular tool. Further, there
was almost no comparisons of the relative effectiveness of pairs
of methods applied to the same task.

%Lacking results from other fields, this study turned to model-based
%methods to assess alternate methods.

%Other researchers have explored meta-heuristic search methods
%(simulated anneling, genetic algorithms, fuzzy logic) for software
%engineering~\cite{seminal01,clark03,harman04}.
%In 2004, Rela~\cite{rela04} reviewed 123 publications in the field
%of search-based software engineering and reported that 44\% were
%related tot esting, while only 7\% of them related to project
%planning. None of that 7\% combined effort and defect and threat
%models or explored the effects of their conclusions on internal
%perturbations to their models.

Lacking results from the literature, we turned to model-based
methods.
This work is an example of the Harman's  {\em search-based
software engineering} (SBSE) paradigm~\cite{harman04,clark03} where SE activities are recast
as optimization problems. 
STAR2 searches COCOMO-family software prediction
models.
Numerous alternate models have been developed by the software prediction
and
modeling community\footnote{
\url{http://www.icsp-conferences.org/icsp2007/}}. Some of these are
specialized models are built to serve the needs of a particular software
development organization. The COCOMO
family of models, on the other hand,
were built to generalize across the community of the COCOMO affiliates.


Historically, this work was inspired by Josephson and Chandrasekaran's 
work on exploration of large design spaces~\cite{joseph98}. In 1999, Josephson 
reviewed our earliest prototype, which was taking too long to terminate. His proposed
solution, 
(try some
stochastic sampling) eventually grew into STAR2.

We have recently become aware that
STAR2's Bayesian ranking methods
is analogous to Ruinstein's cross-entropy method~\cite{rubinstein04}.
Cross-entropy is 
applied during simulated annealing to speed up convergence
to best solutions (by avoiding rare events). 
In future
work, we will explore more  the connection of our Bayesian ranking to
cross-entropy.

Prior related work by this team includes:
\bi
\item
The TAR1  minimal contrast set learner, reported at ASE 2000,
explored 
a Monte Carlo
simulations of the COCOMO and THREAT models~\cite{me00e}.
\item
TAR2~\cite{me03c}, an optimized version of TAR1, 
was applied 
at ASE 2002~\cite{me02f} and RE'02~\cite{fea02a}
to
different software prediction models.
\ei
This paper reports numerous improvements over this prior work:
\bi
\item
Previously, we worked on a single prediction model or pairs of predictive models.
This paper is the first to study effort {\em and} defects {\em and} threats in
one combined analysis.  
\item
This prior work did not adjust the parameters internal
to the prediction model. This new work, on the other hand, makes extensive
adjustments to those parameters.
\item
Previously, our learners commented on just a small subset of the ranges.
This new work ranks all the ranges, allow managers to perform their own detailed
analysis of the relative effects of different project decisions.
\item
This new learner runs much faster than the prior work.
STAR2 generated the OSP and OSP2
results
in 
less than a minute. A similar range ranking,
by TAR1, would 
require an overnight run~\cite{me00e}. TAR2 (and a later version, TAR3)
run much faster but even those optimized systems would
require
20 to 30 minutes to 
generate feature range rankings. 
\item
Not only does STAR2 search faster but it explores a larger space.
As far as we know, 
this is the first report of an extensive exploration of the space of tunings
internal to COCOMO family prediction models.
\ei

\section{Discussion}

We defined ``new ASE tools'' to be:
\[
\begin{array}{c}
\mbox{{\em automated analysis}}\in \{5,6\}\;\vee \\
\mbox{{\em execution-based testing and tools}} \in \{5,6\}
\end{array}
\]
Making the business case for new ASE tools is complicated by
an anti-automation bias ("sociology beats technology") and insufficient data for local tuning of prediction models.

   We hypothesized that precise local tunings are not required to
assess the relative merits of new ASE tools over other methods The
space of tunings is well known- we just need to find what conclusions are stable within that space. Simulated annealing was used to
sample that space and generate a set of 10\% {\em best} and 
90\% {\em rest} solutions
The $N$ ranges in those solutions were ranked using a support 
based Bayesian method. Experiments were performed where the
top $X \le N$ ranged ranges were set and COCOMO, COQUALMO,
and THREAT were allowed to randomly select the remaining ranges.

   By varying the size of $X$, it was possible to find project decisions that most minimized effort/ defect/ threat. Inspecting those
decisions for two NASA systems, we found that new ASE tools
were optional in one (OSP) and essential in the other (OSP2).

   Any generalization based on just two case studies should be treated
with caution. The follow observations should therefore be checked
on other projects:
\bi
\item
 Recall that (a) OSP was an early prototype for OSP2 and
       that (b) new ASE tools were optional for OSP but essential
       for OSP2. That is, {\em project maturity may be a selector for the
       successful use of new ASE tools.}
\item
    In \fig{osp} and \fig{ares}, only very high and extremely high
       range automated analysis or execution-based testing and tools
       were used to minimize effort or defects or threats. Lower
       range usage of these tools was never useful. That is, when
       using new ASE tools, {\em use them thoroughly or not at all}.
    \item In our case studies augmenting sociological decisions with
       new ASE tools can lead to the greatest reduction in effort/
       defects/ threats. That is {\em sociology does \underline{not} beat technology}
       and {\em technology can compliment sociology}.
\item
     When this study could make the business case for new ASE
       tools, a pre-condition for that case was the inclusion of sociological decisions (e.g. about process maturity and schedule pressure) along with
 the new ASE tools. Hence 
{\em ASE researchers need to study both software development technology \underline{and} sociology}.
\ei
   Our conclusions are project-specific but the method of generating them is general to any project that can be described in terms
of the \fig{cparems}
table. An appendix of this paper describes how to
download, install, and use STAR2 to find the most important feature ranges that reduce effort / defects / threats for projects.

{\small
\bibliography{refs}}  % sigproc.bib is the name of the Bibliography in this case
\appendix
\section{Obtaining and Using STAR2}
   These instructions should support a LINUX and CYGWIN install of STAR2.
In the event of technical difficulties, please contact the first two
authors (Menzies or Elrawas).

{\scriptsize
\begin{verbatim}
bash -i
[ ! -d "$HOME/bin" ] && mkdir $HOME/bin
export PATH="$PATH:$HOME/bin"
wget http://unbox.org/wisp/tags/STAR/2.0/STAR_2.0.zip
unzip STAR_2.0.zip
make  # requires gcc
STAR -h # shows all STAR command line options
\end{verbatim}
}

%\section{Results}The results tables are presented here

%\newcommand{\hey}{~~~$\blacktriangle\blacktriangle$}
%This section has the plots.

%\input{strategicallresults}
%\input{strategicflightresults}
%\input{strategicgroundresults}
%\input{strategicospresults}
%\input{strategicaresresults}

%\input{tacticalallresults}
%\input{tacticalflightresults}
%\input{tacticalgroundresults}
%\input{tacticalospresults}
%\input{tacticalaresresults}


\end{document}


