STAR samples the space of possibles models inside COCOMO and COQUALMO using the following 
technique.
Internally, COCOMO and COQUALMO models contain many
linear relationships.
Nominal values of $x=3$ change some estimate by a factor of one.
These COCOMO lines can hence be
 modeled as a straight line $y = mx + b$ passing 
through the point 
${x, y}={3, 1}$. 
Such a line has a y-intercept of 
$b = 1 - 3m$. Substituting this value of $b$ into $y = mx + b$ yields
$y=m(x-3)+1$. COCOMO's effort slopes are either positive or negative, 
denoted $m^+$, $m^-$ (respectively):
\bi
\item
The positive slopes $m^+$
represents the variables that are proportional to effort; e.g.
$increasing$ required reliability also $increases$ the development effort.
\item
The negative
slopes $m^-$
represents the variables that are 
{\em inversely} proportional to effort; e.g.
$increasing$ analyst capability
		 $decreases$ the development effort.
\ei
Based on decades of experiments with calibrating COCOMO models, we have identified
variables with different slopes.
These following COCOMO variables  have $m^+$ slopes:
cplx,
data,
docu,
pvol,
rely,
ruse,
stor, and
time. Also, these variables  have $m^-$ slopes
acap,
apex,
ltex,
pcap,
pcon,
plex,
sced, and
site (for an explanation of those terms,
see \fig{emsf2}). Further, based on decades of calibration of COCOMO models,
we assert that 
 effort estimation, $m^+$ and $m^-$ have the ranges:
\begin{equation}\label{eq:mm}\scriptsize
\begin{array}{c}
 -0.178 \le m^- \le -0.078\\
0.073 \le m^+ \le 0.21
\end{array}
\end{equation}
Using an analogous procedure, it is possible
to derive similar equations 
for  the COCOMO scale factors,
the COQUALMO scale factors/effort multipliers/ defect removal
variables
(for full details, see~\cite{me07f}). 






With the above machinery, it is now possible to define
a Monte Carlo procedure to sample the space of possible
THREAT/COCOMO/COQUALMO $Models$: just
 randomly selecting $\{m^-,m^+\}$.
As to sampling the space of possible
THREAT models, this is achieved by adding  random variables
to the cells of THREAT's  tables.

STAR tries to minimize  defects ($D$), threats ($T$), and 
development effort ($E$).
 This is a non-linear optimization function: e.g. reducing costs can introduce more defects. For this reason, we use simulated annealing (SA) to explore trade-offs between models. 
SA is best explained in comparison to  the Metropolis
algorithm. 


A  {\em Metropolis} Monte Carlo algorithm~\cite{metropolis53}
improves on basic Monte Carlo as follows.
New solutions are created
by  small mutations to some
{\em current} solutions. 
In the case of STAR, an  ``solution'' is some randomly selected
part of the space of possible $Projects$.
If a new solution is ``better'' (as assessed via an
{\em energy function}),  it becomes the new {\em current}
solution used  for future
mutations.  
STAR's energy function is 
\mbox{$E=\sqrt{\overline{E}^2 + \overline{D}^2 + \overline{T}^2} / \sqrt{3}$}
where $\overline{x}$ is a normalized value 
$0 \le \frac{x - min(x)}{max(x) - min(x)} \le 1$. 
Energy ranges $0 \le E \le 1$ and
{\em lower} energies are {\em better}.
If a new solution does not have lower energy, a Boltzmann acceptance
criteria is used to probabilistically decide to
assess the new state: the worse the new state, the less
likely that it becomes the new current state.

A {\em simulated annealer} (SA)~\cite{kirkpatrick83}
adds
a ``temperature'' variable to the Boltzmann accept criteria
such that, at high temperatures, it is more likely that the
algorithm will jump to a  new worst current state.
This allows the algorithm to jump out of local minima while
sampling the space of options. As the temperature cools,
such jumps become less likely and the algorithm reverts
to a simple hill climber.
 
Our {\em RANKER} algorithm instruments the internals of SA.
Whenever a solution is assigned some energy, that energy
is added to a counter maintained
for each variable setting in $Projects$.  When SA terminates,
RANKER
sorts all variable ranges by the sum of the energies seen during
their use. The ranges that are lower in the sort order are associated
with lower energy solutions; i.e. lower defects, efforts, threats.
RANKER then conducts experiments where it fixes the first $N$ ranked
ranges and lets the remaining variables float. $N$ is increased
till some minimum energy point is reached. A {\em policy} are the project
settings that achieve that minimum energy point.

The last two columns of \fig{emsf2} 
show the results of  Delphi panel session at JPL where the COCOMO variables
were separated into those $tactical$ variables
that can be changed within the space of one project,
and those $strategic$ variables
that required higher-level institutional change (and so may take
longer to change). For example, the panel declared that $pmat$ (process maturity)
is hard to change within the space of a single JPL project. 
In the sequel, all our RANKER experiments will be divided into those that
just use the $strategic$ variables and those that just use the $tactical$
variables\footnote{Note that these definitions of $strategic$ and $tactical$ choices
are not hard-wired into STAR. If a user disagrees with our definitions of
strategic/tactical, they can change a simple configuration file.}.

%Ou
%At issue is estimation {\em precision}.
%The less we know about a domain, the greater the uncertainites in the estimates:
%\bi
%\item With no knowledge of effort estimation, then all we can say about
%development effort is that it is a positive number.
%\item With no knowledge of a particular project, all we can say about (say) that project's
%CMM level is that it is between 1 to 5.
%\item With knowledge of the assumptions inside a process model, all we can say is prgrammer
%skill effects development effort (but we do not know the exact nature of that effect).
%\ei
%As we know more about a domain, our estimates move closer to reality. 
%and the 
%probable upper and lower boundaries on that estimate move closer togehter.
%
% 
%\item Project variance dominates tuning variance. That is, we can reduce both the median and variance
%of the output estiamtes 
%\item Project variance dominates
%\ei
%
%At least for the process models discussed below, the effects of project variance dominant tuning
%vDespite this, uncertainties in the effort estiamtes can be tamed by reducing project variance.  
%i
%in the training data, in the model variables ($a,LOC,b,
%  an exponential size has an exponential effect on effort while analyst capa 
%\ei
%
%effort is 
%
%
%Models constrain the 
%\bi
%\item If we adopt the COCOMO assumptions about the relation
%\ei
%
%Our method is {\em stochastic sampling}. Given a model with uncertainties,
%sample across the space of uncertainities looking for stable conclusions.
%In particular, we seek conclusions that reduce 
%estiamted defects, effort, development time, and threats to completion.
%We find that, at least for the models processed here, reducing the median estimates
%also reduces the variance on those estimates. This leads to the following speculation.
%For models where:
%\bi
%\item The goals are minimized;
%\item  Goal variance is less than goal median;
%\ei
%Can 
%\item ... can output variance be tamed using just model inputs?
%\ei
%
%\begin
%
%
%For this experiment, we adopt the following framework.
%We say that a process model $P$ yields estimates from
%a combination of $Project$ and $Model$ variables:
%\[
%estimates = P(Project, Model)
%\]
%$P$ describes the space of influences between variables. 
%	    and may take many forms:
%\bi
%\item
%or the linear models used in COCOMO~\cite{boehm81,boehm00b},
%PRICE-S~\cite{park88} and 
%SEER-SEM~\cite{jensen83}.  
%\ei
%The strength of each influence is controlled
%by the $Model$ variables.
%Taken together, the process model $P$ and the $Model$ variables store
%what we've leaned from the {\em past}. 
%
%$Project$ variables, on the other had,  concern 
%a {\em new} situation that should
%be analyzed using past knowledge.
%For example,
%$P$ could assert  ``$effort \propto pcap$''
%(programmer skills is proportional to development effort)
%while
%$Model$ could assert the proportionality constant of  -0.7
%(i.e. ``$effort=-0.7pcap$'').
%Finally,
%$Project$ could assert that programmer skills are ``in the upper range''; e.g. for a COCOMO model
%``$pcap \in \{4,5\}$''. 
%
%We say $Project$ and $Model$ variables can be:
%\bi
%\item  $fixed$ to one value such as ``programmer capability ($pcap$) is nominal'';
%\item $free$ to take on any legal value. In COCOMO, a free $pcap$ 
%can
%take
%values \[\{veryLow=1, low=2, nominal=3, high=4, veryHigh=5\}\]
%\item or $float$ to some subset of the whole range. 
%For example, a manager might declare that ``our programmers
%are in the upper ranges''; i.e. this
%$pcap$ floats in a particular part of the
%entire $pcap$ range 
%($pcap \in \{4,5\}$).
%\ei
%The range of legal values for variables
%increases from $fixed$ to $float$ to $free$:
%\[\left(|fixed|=1\right) < |float| < |free|\]
%
%This paper reports an experiment that frees {\em both} the $Model$ and
%$Project$ variables.  
%At first glance, such an experiment may seem perverse, particularly if the goal
%is to reduce uncertainty. Free variables range over a larger space
%than fixed variables: the more free variables, the wider the range of $Estimates$. If we free
%both $Model$ and $Project$ variables then, surely, this will result in  greater $Estimate$
%uncertainty?
%
%However, our analysis is not just some passive observer of a large space of options. Instead, it is an active
%agent that seeks parts of the options space where predictions can be made with greater certainty.
%We augment a Monte Carlo analysis with two tools.
%SA is a {\em simulated annealing algorithm} that minimizes   $Estimates$.
%RANKER is a {\em variable pruning algorithm},  that seeks the {\em smallest} number of $Project$ 
%variables that {\em most} reduce the $Estimates$.
%The combination of SA+RANKER is  called STAR\footnote{The name is a geek joke. 
%In regular expressions, the star meta-character ``*''
%matches any characters. That is, just like STAR,  it can be used to search a wide range of options.}.Since
% it  knows
%the most influential $Project$ ranges, 
%STAR can discover (and then constrain)  the factors that most most impact   $Estimates$.  
%
%
%
%\begin{figure}
%{\footnotesize
%
%
%\begin{center}
%\begin{minipage}{.48\linewidth}
%\begin{center}
%\begin{tabular}{|c|r|r|}\cline{2-3}
%\multicolumn{1}{c|}{}&Monte Carlo & STAR\\\hline
%
%case study&SCAT &STAR\\\hline
%
%flight	&712 		&44\\
%ground	&389 		&18	\\
%OSP	&629 		&68	\\
%OSP2	&84		&31	\\\hline
%\multicolumn{3}{c}{~}\\
%\multicolumn{3}{c}{{\bf Fig 1a:} variance, in months.}\\
%\end{tabular}
%\end{center}
%\end{minipage}\begin{minipage}{.48\linewidth}
%\begin{center}
%\begin{tabular}{|c|r|r|}\cline{2-3}
%
%\multicolumn{1}{c|}{~}&
%\multicolumn{1}{|c}{Monte Carlo}&
%\multicolumn{1}{|c|}{STAR}\\\hline
%
%case study &
%\multicolumn{1}{|c}{SCAT}&
%\multicolumn{1}{|c|}{STAR}\\\hline
%flight	&1357 	&86 	\\
%ground	&737	&38 	\\
%OSP	&1951   	&410 	 \\
%OSP2	&297        &182 	\\\hline
%\multicolumn{3}{c}{~}\\
%\multicolumn{3}{c}{{\bf Fig 1b:} median, in months.}\\
%\end{tabular}
%\end{center}
%\end{minipage}
%\end{center}}
%\caption{
%Effort estimates
%seen in 1000 simulations of the $Project$ ranges found by STAR.
%``Variance'' (left hand side)
%shows the difference between the 75th and 50th percentile.
%``Median'' (right hand side)
% shows the 50th percentile estimate. 
% }\label{fig:outv}
%\end{figure}
%
%
%When compared to state-of-the-art process models, the effects
%of a STAR-style analysis are quite dramatic.
%\fig{outv}  compares STAR's estimates to those
% generated by
%SCAT~\cite{lum02,lum03handbook,lum05scat}, a COCOMO-based tool used at
%NASA's Jet Propulsion Laboratory. SCAT fixes $Model$ and perform a
%Monte Carlo simulation of the $Project$ ranges. 
%%% In comparisons with other estimation tools~\cite{lum02}, the SCAT approach
%%% was found to be more accurate than SEER-SEM,
%%% and insignificantly different to PRICE-S.
%%% Lum et.al. prefer SCAT to PRICE-S since, they argue,  SCAT makes it easier
%%% to 
%%% to understand how  specific inputs impact the estimated cost~\cite{lum02}.
%Each row of \fig{outv}.A  is one case study:\bi
%\item $flight$ and $ground$ systems software from
%NASA's Jet Propulsion Laboratory; 
%\item $OSP$ is the GNC\footnote{GNC= guidance, navigation,
%and control} for NASA's Orbital
%Space Plane (prototype); 
%\item $OSP2$ is a newer version of OSP.  
%\ei
% Note that, for all four case
%studies, STAR reduces the variance and median estimates
%to a small fraction of SCAT's
%estimates, sometimes as much as a factor of 20 (in Figure 1a:
%$\frac{712}{44}\approx 20$; in Figure 1b:
%$\frac{737}{38}\approx 20$).
%

This paper
extend prior work in several ways.
A preliminary report on this work reproted the main effect of this paper
(that 
in this paper sometimes 
Prior reports on STAR~\cite{me07f}  were based on  limited case studies; here we report
ten new case studies showing that our main effect (reduced median and variance)
holds in a wide range of cases.
Also, prior reports on  \fig{outv}~\cite{me07g}
failed to check the validity of those results. The ten case studies
discussed below show that STAR's estimated are shown
to be close to those generated via standard local calibration, 
despite being generated
from a large space of $Project$ and $Model$ options.
This validity check greatly increases our confidence in the STAR method.


