STAR is very much a multi-purpose tool that not only has inspired others tools, but was also used in order to explore many problems in the software engineering world. Such problems include the following: 

\bi
\item the importance of automated software engineering (ASE) tools
\item the ability to estimate effort without the presence of model calibration data
\item the efficacy of drastic change versus improving certain aspects of a project with in the projects preset limits
\item addressing the better-faster-cheaper, "pick any two", problem and its implications
\item the future of software engineering and exploring what policies are going to be needed
\ei

Using STAR we will attempt to address these problems and discuss some of the solutions that STAR suggests. 
\section{Sanity Checks}
For the purpose of verifying that our system is valid in its estimations, we need to baseline it against the reference implementations of the models that we are using to make sure that we obtain ballpark estimates at least. For this purpose, a "Sanity" project was created that has all the effort multipliers set to nominal, the scale factors set to extremely high, and the defect removal tools which were left unset to allow STAR to be able to run. Note that these tools do not effect the COCOMO II effort (person-months), the Months (development time), or the Threat estimates. For comparisons with the reference COQUALMO model, the settings for the defect removal was set to the policy produced by STAR. \ref{fig:sanity} presents the results of this exercise, where the last column presents the absolute value of the difference between the Median result that STAR produces (STAR Med) and the reference results, and STAR Sp is the spread of the STAR results. Note that the reference implementations are an online form \footnote{http://sunset.usc.edu/research/COCOMOII/expert\_cocomo/expert\_cocomo2000.html} for COCOMO II, threat and Months and a USC Center of software engineering sourced excel file for COQUALMO.

\begin{figure}
\begin{center}
\begin{tabular}{|l||c|c|c|c|c|}\hline
Model &STAR Med &STAR Sp &Ref &Diff &Diff\%\\\hline\hline
COCOMO II &677.83 &159.75 &840.3 &-162.47 &-19.33\\
COQUALMO &540.39 &313.22 &417.25 &+123.14 &+29.51\\
Months &23.97 &2.52 &24.2 &-0.23 &-0.95\\
Threat &0 &0 &0 &0 &0\\\hline
\end{tabular}
\end{center}
\caption{Sanity Checks}\label{fig:sanity}
\end{figure}

Note that the results for STAR aren't set in stone simply because of the stochastic nature of the search that is conducted with in it, and because of the "wiggling" that is conducted within the model implementations in STAR. Inspecting the results of the above experiment, we can see that the differences aren't large enough to warrant concern, keeping in mind that we aren't adhering to the pre-calibrated models that the reference implementations are. 

\section{Stability and Performance}
In addition to the necessity of verifying that the results of STAR are with in the ballpark of standardized methods, we also need to verify that we have some degree of stability. For this purpose we use two sample projects: one that is highly constrained (OSP) and one that is loosely constrained (flight). STAR was run ten times for both, and then the policy results produced were compared. The table below presents the percentage of times that a certain policy appears: the higher the percentage, the more that is indicative of the stability of that policy.

\input{tables/run-stats}
\input{tables/stability}

\section{STAR NASA}


STAR1 was tested on the four case studies of
%\fig{cases}:
\bi
\item OSP: The Orbital Space Plane GNC prototype (a 1990s NASA system).
OSP was an early prototype for OSP2.
\item OSP2: the guidance and navigation control system of a current NASA launch vehicle, under development.
\item FLIGHT/GROUND: these two are project templates used for nasa projects.
\ei
In \fig{cases},  {\em values} are {\em fixed} while {\em ranges} 
represent a space of options.
\input{figures/cases}
In the sequel, this observation will be important:
OSP2
constrains most of its features to fixed values while
OSP allows more variation in feature ranges. Flight and Ground are more like project templates that, for the most part, enforce restrictions on the ranges of the project. OSP and OSP2 and both considered "Flight" projects.
%\input{tables/braille-nasa}
\input{tables/braille-nasa-collapsed}

\input{figures/nasa/allALLresults}
\input{figures/nasa/strategicALLresults}
\input{figures/nasa/tacticalALLresults}
\input{figures/nasa/allFLIGHTresults}
\input{figures/nasa/strategicFLIGHTresults}
\input{figures/nasa/tacticalFLIGHTresults}
\input{figures/nasa/allGROUNDresults}
\input{figures/nasa/strategicGROUNDresults}
\input{figures/nasa/tacticalGROUNDresults}
\input{figures/nasa/allOSPresults}
\input{figures/nasa/strategicOSPresults}
\input{figures/nasa/tacticalOSPresults}
\input{figures/nasa/allOSP2results}
\input{figures/nasa/strategicOSP2results}
\input{figures/nasa/tacticalOSP2results}



\normalsize
 
 SA was run $K_1$ times and
 ranges not mentioned in \fig{cases}
  selected at random from 
 Equations 2,4,5,7,8 and \tion{threat}.
 The selected {\em controllable} ranges were ranked with
 \eq{ps}:
 \bi
 \item
  By {\em controllable} features, we mean the kinds of changes
 a project manager could make to her project; i.e. \fig{cparems}.
 \item
 {\em Uncontrollable} features
 are features like the correlation between, say, analyst capability and development effort;
 i.e.
 the internal model parameters of 
 Equations 2,4,5,7,8 and \tion{threat}.
 \ei
% \input{ospresults}
% \input{aresresults}
 
 The list at the bottom of \fig{osp}
 shows how \eq{ps} 
  ranked
 OSP project decisions.
 These ranks correspond to the X-axis of the plots at the top of that figure.
 ``Median'' plots the 50\% percentile of the defect/ effort/ threat
 values seen
 after imposing ranges $1..X$ (and selecting all other ranges at random).
 ``Spread'' shows the 75\%-50\% percentile range.
 These plots are U-shaped:
 \bi
 \item On the left-hand-side of each plot,
 poor results were seen
 after applying too few constraints.
 That is, models perform poorly if we do not control them enough.
 \item
 On the right-hand-side of each plot, poor results were also seen.
 In terms of \eq{ps}
 , these are the ranges with low support and
 low probability of belonging to $best$ class. Hence, it is not surprising that
 applying these superfluous constraints was counter-productive.
 \ei
 
 The ranges from new ASE tools (see the ``$\blacktriangle$'' symbol)
 are ranked at X=31, 37, 40, 41.
 These rankings can assessed using the minimum points
 in the median plots:
 \bi
 \item
 At $X\approx15$
 effort and threat are at their global minimums  while
 defects is at a local minimum.
 \item
 The global minimum for defects can be see at $X=33$. 
 At this point, defects are 30\% less than at $X=15$.
 However, this lower defect level is achieved via
 doubling effort
 from 1000 to 2000. months.
 \ei
 
 \fig{osp} shows that OSP needs some tool support.
 The third
 highest ranked range (X=3) is for $tool=3$; i.e.
 medium range tool use,
 This corresponds to some support
 for
 edit, code, and debug
 moderately integrated with processes and methods and reuse. 
 
 However, \fig{osp} 
 makes only a weak business case
 for more extensive tool support.
 {\em Automated analysis} 
 is ranked at positions 40,41 and
 is not required to reach
 the
 minimums of the  \fig{osp} 
plots.  On the other hand,
  if management has the budget
 to support the X=33 effort, then using
 extremely high range {\em execution-based testing  and tools} is 
 strongly recommended. 
 
 Clearly,
 if the project is cost-adverse, then methods other than
 new ASE tools are
 recommended. However, the OSP 
 business users should be aware of the 
 consequences of this policy.
 Applying only decisions $X=1..15$
 will produce
 a product costing half of the previous recommendation,
 but with 30\% more defects.
 
 
 While new ASE tools are {\em optional} for OSP, they are 
 {\em essential} for OSP2.
 \fig{ares}
 shows the OSP2 results.
 Minimum defects were achieved
 at X=17 via a {\em combination} of:
 \bi
 \item {\em technical features}:
 project decisions from the range
 $1 \le X \le 17$
 use
 three of our four new ASE tools (see X=11,14,15).
 \item and {\em sociological features}  such as $docu$,
 use
 of documentation; $pmat$, process maturity; $ltex$,
 language \& tool experience; and
 controlling $sced$ schedule pressure and the $prec$
 precedentedness  of the project.
 \ei
 Hence, the argument quoted in the  introduction that
 ``sociology beats technology''
 should  be corrected to ``sometimes,
 sociology can use more technology''.
 For example, in %\fig{ares}
, only sociological features
 are applied for $1\le X \le 10$. Observe how, at X=11,
 the largest drop in defects was achieved {\em after} applying
 extremely high range automated analysis.
 
 
 Two other noteworthy features 
 of %\fig{ares}
 are:
 \bi
 \item
 The U-shapes seen in all the OSP results of %\fig{osp} 
 only appears in OSP2's defect plot. 
 The median effort and threat curves for OSP2 are essentially flat. 
 As seen in %\fig{cases}
 , OSP2 constrains more
 features than OSP. 
 Hence, at least for OSP2 effort and threats, STAR1 could not change
 those
 median values. 
 \item
 On the other hand,  
 the flatness of the effort and threats plots
 means that they are stable across a wide range of project settings and internal variations to
 the prediction models. For example, according to this study, the effort for OSP2 
 has 50\% to 75\% range of 600 to 900 person months (median
 plus spread). Such stability is very useful for project planning.
 \ei


\section{STAR vs. SCAT vs. 2CEE}


While STAR1 has been shown above to be effective as a recommendation tool for reducing COCOMO effort, defects, and threat, we would like to compare it to other tools that are used for the purpose of reducing COCOMO effort. These tools are SCAT and 2CEE. Note that this comparison isn't exactly an apples to apples comparison, reason being that STAR1 takes a fundamentally different approach compared to those two other tools. While SCAT and 2CEE depend on having a calibrated COCOMO model, STAR1 bypasses that dependency by exploring the space of possible calibrations of the model, and attempting to reduce the COCOMO effort by presenting model recommendations to be applied by the project. It also differs in that it doesn't only work with simply one model, but rather works with a family of models, which in this case is the COCOMO family of models consisting of the Effort, Defects, and Risk models presented in \textbf{insert section reference here}

For this comparison we use the effort results from STAR1 associated with full analysis, along with results from the other two tools after running them on th same projects that STAR1 was run on.


\section{STAR vs. LC}
\input{star-vs-lc}
\subsection{Experiments}

\fig{cases} shows various $Projects$ expressed in term of $float$ing
and $fixed$ variables. For example, with JPL's flight systems,
the $rely$ (required reliability) can float anywhere in the upper range;
i.e. $rely\in\{3,4,5\}$. However, for flight systems, $sced$ (schedule pressure)
is tightly defined (so $sced$ is fixed to the value 3).

\begin{figure}[!t]
\begin{minipage}{.5\linewidth}
\begin{center}
%%%%%%%%%\includegraphics[width=2.4in]{OSP/strategic/Energy.pdf} 
%\includegraphics[width=2.4in]{OSP/strategic/Defects.pdf} 
%\includegraphics[width=2.4in]{OSP/strategic/Effort.pdf} 
%%%%%%%%\includegraphics[width=2.4in]{OSP/strategic/Threat.pdf} 
%%%%%\includegraphics[width=2.4in]{OSP/strategic/Months.pdf} 
{\scriptsize
\begin{tabular}{r|r@{~=~}l}
X& variable & setting\\\hline
1&pmat&4\\
2&ltex&4\\
3&acap&3\\
4&apex&3\\
5&prec&2\\
6&pcon&3\\
7&execution testing and tools&6\\
8&peer reviews&6\\
9&automated analysis&6\\
\multicolumn{3}{c}{}\\
\multicolumn{3}{c}{}\\
\multicolumn{3}{c}{}\\
\multicolumn{3}{c}{}\\
\multicolumn{3}{c}{{\bf Fig.\ref{fig:rrr}.A:}
controlling only strategic $Project$ variables}
\end{tabular}
}
\end{center}
\end{minipage}\begin{minipage}{.5\linewidth}
\begin{center}
%%%%%%%%%\includegraphics[width=2.4in]{OSP/tactical/Energy.pdf} 
%\includegraphics[width=2.4in]{OSP/tactical/Defects.pdf} 
%\includegraphics[width=2.4in]{OSP/tactical/Effort.pdf} 
%%%%%%%%%\includegraphics[width=2.4in]{OSP/tactical/Threat.pdf} 
%%%%%%%%%%\includegraphics[width=3.2in]{OSP/tactical/Months.pdf} 
{\scriptsize
\begin{tabular}{r|r@{~=~}l}
X& variable & setting\\\hline
1&ruse&2\\
2&cplx&5\\
3&resl&3\\
4&tool&3\\
5&sced&2\\
6&stor&3\\
7&flex&5\\
8&automated analysis&6\\
9&peer reviews&6\\
10&docu&2\\
11&execution testing and tools&6\\
12&sced&1\\
\multicolumn{3}{c}{}\\
\multicolumn{3}{c}{{\bf Fig.\ref{fig:rrr}.B:}
controlling only tactical $Project$ variables}
\end{tabular}
}
\end{center}
\end{minipage}
\caption{Some RANKER results on OSP. The settings shown under the plots
describe the policy that leads to the policy point.}\label{fig:rrr}
\end{figure}
\begin{figure}
\begin{minipage}{.5\linewidth}
\begin{center}
%%%%%%%%\includegraphics[width=2.4in]{ground/strategic/Energy.pdf} 
%\includegraphics[width=2.4in]{ground/strategic/Defects.pdf} 
%\includegraphics[width=2.4in]{ground/strategic/Effort.pdf} 
%%%%%%%\includegraphics[width=2.4in]{ground/strategic/Threat.pdf} 
%%%%%%%\includegraphics[width=2.4in]{ground/strategic/Months.pdf} 
{\scriptsize
\begin{tabular}{r|r@{~=~}l}
X& feature & range\\\hline
1&pmat&3\\
2&site&6\\
3&pcon&5\\
4&plex&4\\
5&pcap&5\\
6&ltex&4\\
7&apex&5\\
8&prec&5\\
9&acap&5\\
10&automated analysis&6\\
11&execution testing and tools&6\\
12&peer reviews&6\\
13&acap&4\\
\multicolumn{3}{c}{}\\
\multicolumn{3}{c}{{\bf Fig.\ref{fig:yyy}.A:}
controlling only strategic $Project$ variables}
\end{tabular}
}
\end{center}
\end{minipage}\begin{minipage}{.5\linewidth}
\begin{center}
%%%%%%%\includegraphics[width=2.4in]{ground/tactical/Energy.pdf} 
%\includegraphics[width=2.4in]{ground/tactical/Defects.pdf} 
%\includegraphics[width=2.4in]{ground/tactical/Effort.pdf} 
%%%%%%\includegraphics[width=2.4in]{ground/tactical/Threat.pdf} 
%%%%%%\includegraphics[width=2.4in]{ground/tactical/Months.pdf} 
{\scriptsize
\begin{tabular}{r|r@{~=~}l}
X& feature & range\\\hline
1&resl&5\\
2&cplx&1\\
3&execution testing and tools&6\\
4&flex&5\\
5&docu&1\\
6&ruse&2\\
7&data&3\\
\multicolumn{3}{c}{}\\
\multicolumn{3}{c}{}\\
\multicolumn{3}{c}{}\\
\multicolumn{3}{c}{}\\
\multicolumn{3}{c}{}\\
\multicolumn{3}{c}{}\\
\multicolumn{3}{c}{}\\
\multicolumn{3}{c}{{\bf Fig.\ref{fig:yyy}.B:}
controlling only tactical $Project$ variables.}
\end{tabular}
}
\end{center}
\end{minipage}
\caption{Some RANKER  results on JPL ground systems.
The settings shown under the plots.
describe the policy that leads to the policy point.
}\label{fig:yyy}
\end{figure}


\fig{rrr} and \fig{yyy}
shows the results of STAR.
The variable ranges are sorted along the x-axis according the order
generated by RANKER. 
At any $x$ value we see the results of fixing the ranges $1..x$, letting
all ranges $x+1... max$ float,
then
running 1000 Monte Carlo simulations. In the results, ``median'' refers
to the 50th percentile band and ``spread'' refers to the difference
between the 75th and 50th percentile in the 1000 generate estimates.


For this paper, we ran SA+RANKER on the four case studies
of \fig{cases}, plus a fifth study called ``ALL'''' that
used the entire COCOMO ranges,
unconstrained by a particular project. Each study was repeated
twice- one for controlling just the strategic variables
and once for controlling just the tactical variables.
This resulted in ten experiments.

Some of the results from four of those experiments are shown in \fig{rrr}
and \fig{yyy} (space restrictions prevent us from showing all the results).
In those four
experiments
 (and in the other six,  not shown) the same effect was observed.
Minimum effort and defects was achieved after fixing
a small number of $Project$ variables
(in 
 \fig{rrr}.A, \fig{rrr}.B, \fig{yyy}.A, and \fig{yyy}.B,
that number was
at X=\{9,12,13 7\} respectively).
 At these minimum
points, the median and spread estimates were greatly reduced.
We call this minimum the $policy point$ and use
the term $policy$ to refer
to  the intersection of the case study defined in \fig{cases},
and the ranges found in the range between 
$\{1 \le x \le policy point\}$.

\fig{rrr} and \fig{yyy} are the reports we would offer back to the
manager.  Start at the top of this list, we would advise, and apply
as many oft eh top $N$ things that you can. Do not waste time
implementing policy changes off this list.

In terms of controlling
uncertainty, the reduction in the spread estimates
at the policy point
is particularly interesting.
Note that this reduction in model uncertainty
was achieved by only
controlling a few of the $Project$ variables while
letting all other $Project$ and $Model$ variables float free.
That is, in these case studies, projects could be controlled
(development effort and defects reduced) without using
historical data to constrain the $Model$ variables.

For each of our ten experiments,
a set of random $Projects$ were generated, consistent
with the policies; i.e. 
\bi
\item
If the policy fixes a value, then the $Project$ contains that value;
\item
Otherwise, if the variable is found \fig{cases}, it is drawn from those constraints;
\item
Otherwise, the variable's value is selected at random from background knowledge of the legal range of the \fig{emsf2} variables.
\ei
For each set, the following procedure was repeated 20 times.
Ten examples were removed at random and Boehm's local calibration (LC) procedure~\cite[p526-529]{boehm81}  was
used to train a COCOMO model on the remaining $Project$ 
examples\footnote{LC was chosen since, in extensive experiments,
we have found this decades old procedure to be 
remarkably competitive with current data mining methods~\cite{jalali07}
including bagging and boosting~\cite{baker07}.}. LC's estimates were then compared to the estimates generated by
STAR's simulation at the policy point (i.e. floating over both the policy and the $Model$ ranges). 
\fig{deltas} show the median difference in the 
 estimates generated
by LC or STAR . Note that, in $\frac{8}{10}$
cases, the difference is under 50\%. 
The reason for the large deltas seen in $\frac{2}{10}$ of the
results (from the OSP case study)
are currently unknown but
are a subject of much current exploration.
%\begin{figure}
%\begin{center}
%\includegraphics[width=1.5in]{graphs/OSP2-tactical.pdf}    %  34.8
%\includegraphics[width=1.5in]{graphs/ALL-strategic.pdf}    %  35
%\includegraphics[width=1.5in]{graphs/OSP2-strategic.pdf}   %  35.1
%\includegraphics[width=1.5in]{graphs/flight-tactical.pdf}  %  35.6
%\includegraphics[width=1.5in]{graphs/ground-tactical.pdf}  %  37.3
%\includegraphics[width=1.5in]{graphs/ALL-tactical.pdf}     %  41
%\includegraphics[width=1.5in]{graphs/flight-strategic.pdf} %  42.1
%\includegraphics[width=1.5in]{graphs/ground-strategic.pdf} %  49
%\includegraphics[width=1.5in]{graphs/OSP-tactical.pdf}     % 112.1 
%\includegraphics[width=1.5in]{graphs/OSP-strategic.pdf}    % 146.5
%\end{center}
%\caption{Effort estimates generated by LC (thin line) and 
%STAR (fat line). The ten plots are sorted in order of the median differences
%(shown in \fig{deltas}. Within in each plot,
%the results are sorted by the size of estimated offered by LC
%(the thin line).
%}\label{fig:group1}
%\end{figure}

\begin{figure}[!t]
\begin{center}
\scriptsize
\begin{tabular}{l|c|rl}
cast study&~control method &~~ $\delta$ \%  & \\\hline
OSP2&tactical&34&\rule{15mm}{2mm}\\
All&strategic&35&\rule{15mm}{2mm}\\
OSP2&strategic&35&\rule{15mm}{2mm}\\
flight&tactical&36&\rule{18mm}{2mm}\\
ground&tactical&37&\rule{18mm}{2mm}\\
All&tactical&41&\rule{20mm}{2mm}\\
flight&strategic&42&\rule{21mm}{2mm}\\
ground&strategic&49&\rule{24mm}{2mm}\\
OSP&tactical&112&\rule{56mm}{2mm}\\
OSP&strategic&147&\rule{74mm}{2mm}
\end{tabular}
\end{center}
\caption{Median 
 $\delta=(estimate(STAR) - estimate(lc))$
between effort
estimates generated by conventional means (LC)
and STAR.}\label{fig:deltas}
\end{figure}


The median $\delta$ values of \fig{deltas} are around 
0.4; i.e. a STAR estimate of 100 months
could really range for 60 to 140 months.
Compared to
the effort estimate reductions shown in the  introduction, $\delta$ 
is quite small.
Recall that STAR reduced effort estimates to a small part of the initial values, sometimes
a factor of 20; i.e by  a factor that is much larger than 0.4. Clearly,
even if STAR is wrong by $\pm 40$\%, then the overall benefits to be gained from applying
STAR's policies are  still
dramatically large.

 % ---- Bibliography ----
%

\subsection{Discussion}
Given all the randomized exploration
STAR performs over the space of possible $Models$, this discrepancy is very small.
and 
those discrepancies are dwarfed 
by the much larger  effort reductions of Figure 1.

How are we to  explain the remarkable effectiveness of STAR in
managing uncertainty?
Researchers in planning and theorem proving have
recently shown that as model complexity grows, other constraining
effects may appear such as ``master variables''; i.e. a small number
of settings that control all other
settings~\cite{crawford94,williams03}. Such master variables can
greatly reduce the search space within large models.  

 We hypothesize that software process models also contain master variables;
i.e. much 
 much of uncertainty in a model is due to the influence of a 
small subset
of model variables. If so, then after (a)~ranking variables
by their ability to constrain the output; and
(b)~applying a small number of the top-ranked variables;
then 
it should be possible to (c)~make
stable predictions
in the constrained space.
\subsection{Conclusion}


In studies with one widely-used suite of effort/ detect/ threat predictors 
for software systems, we have shown that:
\bi
\item 
$Estimation$ median values can be greatly reduced (see Figure 1). In comparisons with other effort
estimation tools, the reduction can quite dramatic. In the best case our tools found
$Project$ ranges that yields estimates that were 5\% of estimates found by other means.
\item $Estimation$ variance can be reduced by only floating the $Project$ values and leaving the $Model$ values free (see \fig{rrr} and \fig{yyy}).
\item
Within the space of $Project$ options that most reduce $Estimation$ median and variance, the
predictions made by our process models are remarkably similar to those made by 
conventional methods (see \fig{deltas} ). 

\ei


The
first result suggests that it may be highly advantageous to use STAR.
Projects designed around STAR's recommendations will be
will be
delivered sooner and have fewer bugs or threats.


The second result is of much practical importance  since it means
we do not require calibration data to tune the $Model$ variables. 
If process models can be deployed without calibration, then they
can be used with much greater ease and {\em without} the requirement for an expensive and time-consuming period
of data collection.

The third result is showing that (a)~this method can find and remove the major sources of uncertainty in a project;
(b)~in the reduced space, it is possible that the estimates
in the resulting constrained space will be close
to estimates found via tuning on historical data.
In the above discussion section, we commented that this result
has precedent in the AI planning and  theorem proving literature.

Finally, we comment on the external validity of these results.
Compared to many other process models\footnote{See
 Software Process journal,  issue on Software Process Simulation,
vol. 7, No. 3-4,  2002.} this combination of  effort/threat/defect models is relatively simple.
As model complexity grows, then the space of possible $Estimates$ can grow exponentially and STAR's
controlling effect may disappear. 
Therefore it is clear that   we can not claim that, for {\em all}
process models, that
$Estimate$ variance can be controlled by just constraining $Project$, not
$Model$, variance.

Nevertheless, data collection for the purposes of model calibration remains
as  a
expensive, tedious, and often incomplete process.
Our results suggest that such data collection may be, for some process models,
an optional activity (caveat: provided that a process model exists that
specifics the general relationships between concepts in a domain).
Our hope is that the results of this paper
encouraging enough that other software process modeling researchers
will try the following strategy:
\bi
\item finding the fewest number of variables that most effect model output;
\item constrain them;
\item check for stable conclusions in the constrained space.
\ei
If these results from STAR generalize to more complex models,
then is should be possible to make reasonably accurate
predictions without local calibration data.

Note that if such stability is {\em absent} in more complex models,
and those models are being used in domains with data collection problems,
then we would argue that that is a reason to abstain from such complexity,
and use COCOMO-style models instead.


\section{JPL Studies}
%\input{tables/jpl-braille}
\input{figures/JPLcases}
\input{tables/jpl-braille-collapsed}

\section{Futures}

\subsection{Data Collection}
In this section we will describe how data was collected in order to provide some insight on future trends in the software industry. That will be followed with presenting the results of this data collection and how those were used in STAR2.

\begin{figure}[!t]
\begin{center}
\begin{tabular}{cc}
\includegraphics[scale=0.45]{misc/workshop1.png}&\includegraphics[scale=0.45]{misc/workshop2.png}
\caption{Pictures of the panel.}\label{fig:pan}
\end{center}
\end{figure}


\scriptsize
\begin{figure}
\begin{center}

\begin{tabular}{l|c|l|c}
feature&setting&feature&setting\\\hline
aa&3&tool&3\\
acap&3&time&3\\
apex&4.5&team&3.5\\
cplx&3&stor&3\\
data&4&site&4.5\\
docu&3&sced&3\\
ett&2&ruse&4\\
flex&3&resl&3\\
ltex&4.5&rely&2.5\\
pcap&3&pvol&2\\
pcon&3&prec&4.5\\
peer&2.5&pmat&2.5\\
plex&4.5\\
\end{tabular}
\end{center}
\caption{The baseline settings representing present day software engineering}\label{fig:base1}
\end{figure}



\begin{figure}
\begin{center}
\begin{tabular}{lrr|lr}
       &\multicolumn{1}{c}{ranges}      &\multicolumn{3}{c}{values}\\\hline
feature&low&high&feature&setting\\\hline
aa&3&6&data&4\\
acap&3&5&docu&3\\
apex&1&4.5&ltex&4.5\\
cplx&3&6&pcap&3\\
ett&2&6&plex&4.5\\
flex&3&5&team&3.5\\
pcon&1&3&&\\
peer&2.5&6&&\\
pmat&2.5&5&&\\
pvol&2&5&&\\
rely&2.5&5&&\\
resl&3&5&&\\
ruse&2&6&&\\
sced&1&3&&\\
site&1&4.5&&\\
tool&3&5&&\\
\end{tabular}
\end{center}
\caption{Common settings for all future trend variants}\label{fig:cases1}
\end{figure}

\normalsize

The panel meeting was conducted during ICSE 2008 on May 13. \fig{pan}
 are photos of this panel.
It was composed of eight seasoned experts in the field of software engineering:
\bi
\item
Barry Boehm
\item
Vic Basili
\item
Ray Madachy
\item
Thomas Ostrand
\item
Deiter Rombach
\item
Rick Selby
\item
Elaine Weyeuker
\ei

These experts formed the part of the panel from whom insights concerning the future trends in the software industry was derived. They discussed projections concerning all the COCOMO II/COQUALMO factors. Note that the time frame for these projections is within a decade. Other assumptions concerning the future projects include the following:
\bi
\item
The projects/applications are {\em NASA-like} applications.
\item
They are {\em mission critical real-time} applications.
\item
They are {\em contractor built}.
\item
They have a {\em long life time} (between 5 and 10 years).
\ei


Three facilitators were present as well:
\bi
\item
Bojan Cukic
\item
Tim Menzies
\item
David Raffo
\ei

The role of the facilitators was to guide the discussion on the model attribute projections in an attempt to keep the discussions focused. Gregory Gay acted as scribe during this session in order to transcribe notes from the meeting.

During discussions, consensus was rarely reached concerning projections about how the software environment might change relative to the models. For this reason, after brief discussions about every attribute, a vote was taken concerning the future behavior of the attribute. These behaviors include the following:
\bi
\item
\textbf{Bump}: The attribute would increase in value at first, and then decrease later on. PREC was voted to have such behavior.
\item
\textbf{Rick}: This is the opposite of bump, where attribute would decrease in value at first, and then increase later on.
\item
\textbf{Up}: The attribute would increase in value. ACAP was voted to have such behavior.
\item
\textbf{Down}: The attribute would decrease in value. APEX was voted to have such behavior.
\ei

Some chose to abstain from some of the votes as well. %\fig{votes1} and \fig{votes2} 
contain the summary of the results of these discussions, as well as the votes.

Note that in order to have future projections, we had to set a baseline that represent the current trends in software engineering. This had been set prior to the panel session with input from one of the expert panelists, Barry Boehm, and is presented in %\fig{base}
. From this baseline, we branched 8 main projections. the reason being that, as we see in %\fig{votes1} and \fig{votes2}
, only 3 attributes (PREC, STOR and TIME) were voted as not having a single direction of variation, namely bumps. %\fig{cases}
 shows all the other attributes that were voted to have a single direction of change, and there corresponding ranges and values.

\subsection{policies}

When referring to %\fig{exlarge}, \fig{exmedium} and \fig{exsmall}
, we notice that the policies that are chosen by STAR2 are very similar across project sizes. Exceptions to this exist, such as the tendency of STAR2 to prefer larger values for SCED for larger projects when compared to the results of medium and small projects. This suggests that, for larger projects, the cost of speeding up a project schedule will  probably out way the benefits, as this might increase the amount of defects that end up in that large project, which would cost the project additional effort later on in the project life cycle. Keep in mind that one of our assumptions for the projects is that they possess a longer lifetime, which would emphsize such a project disadvantage.

Another effect that we observe is that there isn't as much variation between the policies of separate projections as we would expect. This would seem to suggest that, even though we have 8 separate base projections, the overall projection is stable and determinant enough to actually draw stable conclusions from that would apply across the board. In other words, for the purposes of STAR2, the experts panel from whom the projections originate don't necessarily need to have consensus for us to draw stable future predictions about policies that need to be applied in the future.

The effect that most interests us for the purposes of this article is related to ASE tools. On the one hand these tools are recommended for the majority of our case studies. This applies most to those cases with full or technical analysis, and to a lesser extent to those with strategic analysis. Note however that in all cases, whenever these tools are applied, they are applied at very high or extremely high levels 
(between 5 and 6), which emphasizes the importance of advanced ASE tools for future projects. These tools however have the most utility whenever the project is in the development phases. This explains our result since statistical analysis is more related to the planning stages of a project, which tactical analysis is more related to the development stage of a project. Also notice that for the same analysis methods the emphasise the use of advanced ASE tools, the TOOL is recommended at its highest value, which means having it integrated in to the life cycle according to \ref{fig:cparams}
. So there seems to be a correlation between having the need for ASE tools, and requiring having these Tools integrated into the software life cycle.

\subsection{Reductions}
The lower parts of %\fig{exlarge}, \fig{exmedium} and \fig{exsmall} 
refer to the size of the final model prediction values compared to the initial ones, i.e. the ones before applying any kind of policies on the case study in question. So smaller is better. We notice four main effects in these tables.

The first effect is that the reductions are very similar across case studies of the same kind of analysis. This doesn't apply in all cases however. The exceptions include such cases were ASE tools aren't applied at all. These tend to have much lower reductions in defects compared to other cases with these tools were being applied. However, the overall effect seems to be that overall the reductions are similar for the same analysis types regardless of the projection or project size. This is mainly due to the similarity in the policies that are followed, which is mentioned in the previous section.

Secondly, effort reductions seem to be larger for strategic compared to tactical analysis. This would suggest that strategic long term planning is more effective at reducing cost for future projects. Note that our strategic analysis with STAR2 didn't decisively recommend the use of advanced ASE tools. So there seems to be an argument that ASE tools are necessary for cost reduction. However this isn't the whole picture.

We also observe that defect reductions are largest with tactical analysis. This is due to the affinity of this kind od analysis to recommend ASE tools be used. ASE tools are mainly defect reduction tools, so its only natural that the analysis that most recommends these tools achieves higher defect reductions. The fact that its tactical analysis suggests that ASE tools are best used during the development stages of a software product rather than during the planning stages.

The last effect that we observe is that full analysis achieves the best reductions by a large margin. This applies for all the models and all the case studies, suggesting that if we could control everything in a project, we should. This unfortunately isn't really possible nor feasible. However this can be interpreted as using both proper strategic and tactical planning when appropriate. So, use strategic planning on a future project, apply the policies, then transition to tactical planning and apply the associated policies. By doing this, a software project could achieve a similar effect to controlling all, or most, of the variables in a software project achieving high reductions in effort, defects, and months (schedule). Note that this type of analysis recommends used advanced ASE tools most of the time, providing a strong business case for them.

Note that the threat model doesn't seem to have much importance in our case studies in that model never seem to register any tangible values in any of the experiments. This might be because of some bug in the implementation of the model itself or in our reporting methods and scripts. However, it is more likely to be caused by the nature of the case studies themselves, since they aren't very constrained case studies. We have seen this before, where for less constrained case studies threat doesn't really come into play.

\begin{center}
\input{tables/icseMay13}
%\input{tables/futures-braille-1}
\input{tables/futures-braille-2}
\input{tables/futures-summary}
\end{center}
\section{Better, Faster, Cheaper}
\input{figures/bfc/winloss-bfc}
\section{Drastic vs. Conservative Control (Hoh In strategies)}
\input{figures/hohin/winloss-hohin}
