% This is "sig-alternate.tex" V1.7 MARCH 2007
% This file should be compiled with V2.1 of "sig-alternate.cls" MARCH 2007
%
% This example file demonstrates the use of the 'sig-alternate.cls'
% V2.1 LaTeX2e document class file. It is for those submitting
% articles to ACM Conference Proceedings WHO DO NOT WISH TO
% STRICTLY ADHERE TO THE SIGS (PUBS-BOARD-ENDORSED) STYLE.
% The 'sig-alternate.cls' file will produce a similar-looking,
% albeit, 'tighter' paper resulting in, invariably, fewer pages.
%
% ----------------------------------------------------------------------------------------------------------------
% This .tex file (and associated .cls V2.1) produces:
%       1) The Permission Statement
%       2) The Conference (location) Info information
%       3) The Copyright Line with ACM data
%       4) NO page numbers
%
% as against the acm_proc_article-sp.cls file which
% DOES NOT produce 1) thru' 3) above.
%
% Using 'sig-alternate.cls' you have control, however, from within
% the source .tex file, over both the CopyrightYear
% (defaulted to 200X) and the ACM Copyright Data
% (defaulted to X-XXXXX-XX-X/XX/XX).
% e.g.
% \CopyrightYear{2007} will cause 2007 to appear in the copyright line.
% \crdata{0-12345-67-8/90/12} will cause 0-12345-67-8/90/12 to appear in the copyright line.
%
% ---------------------------------------------------------------------------------------------------------------
% This .tex source is an example which *does* use
% the .bib file (from which the .bbl file % is produced).
% REMEMBER HOWEVER: After having produced the .bbl file,
% and prior to final submission, you *NEED* to 'insert'
% your .bbl file into your source .tex file so as to provide
% ONE 'self-contained' source file.
%
% ================= IF YOU HAVE QUESTIONS =======================
% Questions regarding the SIGS styles, SIGS policies and
% procedures, Conferences etc. should be sent to
% Adrienne Griscti (griscti@acm.org)
%
% Technical questions _only_ to
% Gerald Murray (murray@acm.org)
% ===============================================================
%
% For tracking purposes - this is V1.7 - MARCH 2007

\documentclass{sig-alternate}
\newcommand{\aive}{{\"aive}}

%-- Begin patch area for accents in 'Author Block' area - may be needed by some authors / but not all
\DeclareFixedFont{\auacc}{OT1}{phv}{m}{n}{12}   % Needed for "Author Block" accents - Patch by Gerry 3/21/07
\DeclareFixedFont{\afacc}{OT1}{phv}{m}{n}{10}   % Needed for "Author Block" accents in the affiliation/address line - Patch by Gerry 3/21/07
%--
\usepackage{color,cite,tricks}
\usepackage{url,amssymb}
\DeclareMathOperator*{\argmin}{arg\,min}
\newcommand{\hmm}[1]{{\underline {\bf #1}}}
\begin{document}
%
% --- Author Metadata here ---
\conferenceinfo{ASE}{'07 Atlanta, Georgia, USA}
%\CopyrightYear{2007} % Allows default copyright year (200X) to be over-ridden - IF NEED BE.
%\crdata{0-12345-67-8/90/01}  % Allows default copyright data (0-89791-88-6/97/05) to be over-ridden - IF NEED BE.
% --- End of Author Metadata ---

\title{The Business Case for Automated Software Engineering}
\numberofauthors{3} %  in this sample file, there are a *total*
% of EIGHT authors. SIX appear on the 'first-page' (for formatting
% reasons) and the remaining two appear in the \additionalauthors section.
%
\author{ 
% You can go ahead and credit any number of authors here,
% e.g. one 'row of three' or two rows (consisting of one row of three
% and a second row of one, two or three).
%
% The command \alignauthor (no curly braces needed) should
% precede each author name, affiliation/snail-mail address and
% e-mail address. Additionally, tag each line of
% affiliation/address with \affaddr, and tag the
% e-mail address with \email.
%
% 1st. author
\alignauthor
Tim Menzies\\Oussama Elrawas\\
       \affaddr{Lane Dept. CSEE}\\
       \affaddr{West Virginia University, USA}\\
       \email{tim@menzies.us}\\
	\email{oelrawas@mix.wvu.edu}
% 2nd. author
\alignauthor
Jairus Hihn\\Martin Feather\\
       \affaddr{Jet Propulsion Laboratory}\\
       \affaddr{California, USA}\\
       \email{mfeather@jpl.nasa.gov}\\
       \email{jairus.hihn@jpl.nasa.gov}
\alignauthor
Ray Madachy\\Barry Boehm\\
       \affaddr{Computer Science Dept. }\\
       \affaddr{Uni. Southern California, USA}\\
       \email{madachy@usc.edu}\\
       \email{boehm@sunset.usc.edu}
}
      % There's nothing stopping you putting the seventh, eighth, etc.
% author on the opening page (as the 'third row') but we ask,
% for aesthetic reasons that you place these 'additional authors'
% in the \additional authors block, viz.
\date{\today}
% Just remember to make sure that the TOTAL number of authors
% is the number that will appear on the first page PLUS the
% number that will appear in the \additionalauthors section.

\maketitle
\begin{abstract}
In theory, software process models
can 
assess the relative merits of automated software engineering tools.
In practice, such an assessment is complicated by the 
{\em tuning problem}; i.e.
it may be  hard
to collect the local data required for model tuning. 
However, while the exact tunings may not be available, the space of
possible tunings is well known.
For two NASA case studies,
we explore that space to conclude that
automated software engineering tools are essential for reducing
project
effort and defects and threats
\end{abstract}

% A category with the (minimum) three required fields
\category{I.6}{Learning}{Machine Learning}
%\category{D.2.9}{Software Engineering}{Effort Estimation}
%A category including the fourth, optional field follows...
\category{D.2.8}{Software Engineering}{Metrics}[product metrics, process metrics]


\keywords{COCOMO, COQUALMO, meta-heuristic search, simulated annealing, feature subset selection}
\pagestyle{plain}
\section{Introduction}

Automated software engineering (ASE) is one of the most successful
experiments in the history of computer science. Decades of research
has yielded tools and techniques that allow even a single individual
to built a robust operating system (e.g. Torvalds' LINUX
kernel) or even innovative programming languages 
that can attract
a world-wide following (e.g. Wall's PERL or 
Matsumoto's RUBY).

Having solved the {\em generation} problem 
ASE research has now moved
on to the management and  maintenance problems of 
systems that are larger or more safety critical.
Much ASE research focuses on 
the {\em automatic analysis} of source code or the better
{\em execution-based testing tools},
These tools might, for example,
verify formal properties or
search for the fewest regression tests that exercise  most of the system.
Many of these new
tools are ready for industrial use; e.g, SPIN~\cite{holzmann97} or 
JPF~\cite{have00}, just to name
a few.   

Making the business case for using this next generation of
 high-end ASE tools
is not an easy task.
\bi
\item
At a recent
panel at ICSE 2007, Tim Lister (one of the authors of Peopleware~\cite{demarco87})
commented
that ``sociology beats technology
in terms of successfully completing projects'';
project managers who manage the sociology of their people (e.g. to decrease
staff turnover) do better than managers that stake the success of their
project on some supposedly superior tool.
\item
\fig{coceffects} shows the known relative productivity effects
of changing project features (data from
a regression analysis of 161 projects~\cite{boehm00a}).  
\fig{coceffects}
support Lister's thesis that 
the benefits of automatic tools (number nine in the list) can be equaled or bettered
via other means; e.g. any {\em one} the first eight items or any {\em pair} of items 
10 to 22.
\ei
\begin{figure}[!b]
\small
\begin{center}
\begin{tabular}{l@{}|l@{~}r@{~}l}
& & relative\\
id & features  & weight\\\hline
1&Personnel/team capability  & 3.53	&\rule{17mm}{2mm}\\
2&  Product complexity & 2.38&	\rule{12mm}{2mm}\\
3& Time constraint & 1.63&	\rule{8mm}{2mm}\\
4&    Required software reliability & 1.54&	\rule{8mm}{2mm}\\
5&    Multi-site development   &	1.53&	\rule{7mm}{2mm}\\
6&       Doc. match to life cycle  & 	1.52	&\rule{7mm}{2mm}\\
7&  Personnel continuity  &	1.51&	\rule{7mm}{2mm}\\
8&  Applications experience & 1.51	&\rule{7mm}{2mm}\\
9& Use of software tools & 1.50	&\rule{7mm}{2mm}\\
10& Platform volatility & 1.49&	\rule{7mm}{2mm}\\
11& Storage constraint & 1.46&	\rule{7mm}{2mm}\\
12& Process maturity  &	1.43	&\rule{7mm}{2mm}\\
13&    Language \& tools experience  &	1.43	&\rule{7mm}{2mm}\\
14&     Required dev. schedule & 1.43	&\rule{7mm}{2mm}\\
15&Data base size & 1.42&	\rule{7mm}{2mm}\\
16&  Platform experience & 1.40&	\rule{7mm}{2mm}\\
17&     Arch. \& risk resolution  &	1.39&	\rule{7mm}{2mm}\\
18&  Precedentedness  &	1.33&	\rule{6mm}{2mm}\\
19&  Developed for reuse  	&1.31	&\rule{6mm}{2mm}\\
20& Team cohesion  &	1.29&	\rule{6mm}{2mm}\\
21&    Development mode   &	1.32	&\rule{6mm}{2mm}\\
22& Development flexibility &	1.26	&\rule{6mm}{2mm}\\
\end{tabular}
\end{center}
\caption{Relative effects  on development effort.
}\label{fig:coceffects}
\end{figure}
\input{cocomo}

One way to make the business case for high-end ASE tools is to argue
that the real benefit of these tools is that they generate
high quality software (e.g. with fewer defects). \fig{coceffects}
only concerns itself with development effort and, hence, is blind
to the quality improvement benefits of high-end ASE tools.
A complete
assessment of high-end ASE tools
must 
explore the relative value of automated tools
for
minimizing development effort {\em and} the  defects
delivered in the system, 
{\em and} any  other threats to the successful completion of the
project. 
There are  many
predictors for these qualities including the COCOMO-family
of models:
\bi
\item
The COQUALMO defect predictor~\cite[p254-268]{boehm00b};
\item
The  COCOMO effort predictor~\cite[p29-57]{boehm00b}; 
\item
The THREAT  schedule threat predictor~\cite[284-291]{boehm00b}.
\ei
These models are defined using 
the features of \fig{cparems}.
Note that high-end ASE tools appear in the right-hand column of the first two
rows of that figure.

When {\em tuned} to local data, COQUALMO, COCOMO, and THREAT
are quite accurate predictors (see \tion{ev}).
Unfortunately, {\em the data required for the local tuning may be incomplete}.
Software projects are notoriously difficult
to control: recall the 1995 report of the Standish group
that described a \$250 bullion dollar American software industry
where 31\% of projects were canceled and 53\% of projects incurred
efforts exceeding 189\% of the original estimate~\cite{standish95}.
Corporations are therefore reluctant to expose their own software
development record to public scrutiny.
For example, despite two years intensive effort, the authors have failed
to obtain local tuning data from NASA.

The premise of this paper is that, in terms of ranking different methods,
a {\em precise} local tuning is not required.
Based on research dating back to 1981~\cite{boehm81}, we assert that the
space of {\em
possible} tunings is well known. This space can be explored to
find features that minimize effort, defects, and threats.  

The STAR 1.0 algorithm, developed for this paper,
adds a novel
back select algorithm, to a simulated annealer.
STAR 1.0
was used to  explore the space of possible tunings within
COCOMO, COQUALMO and THREAT. 
SA
offers constraints to all variables. STAR 1.0 ranks and prunes those
constraints. At each step in the pruning, the COCOMO/ COQUALMO/
THREAT models are called to test the current  constraint set in a series of simulations.
The result is an ordering on the feature ranges according to how
well they reduce
effort/ defects/ threats.
In terms of the business case for automated software engineering, the important
result will be that, for NASA systems,
high-end ASE tools were found 
in the minimum set of projects constraints
that reduced effort/ defects/ threats.  


The rest of this paper documents the local tuning problem for
the 
COCOMO, COQUALMO,
and THREAT models, along with the STAR 1.0 algorithm.
STAR 1.0 is then applied to two NASA case studies.
This will be 
followed by notes on external validity and related work.,
\section{The Local Tuning Problem}\label{sec:inc}
%\subsection{Inherent Bias?}
%
%As mentioned above, our research funding depends on the continued interest
%of funding agencies in our work.
%It might hence  be argued that the authors are the {\em worst} team to comparatively
%evaluate automatic tools vs manual methods.
%It is therefore prudent to question our
%evaluation bias.
%
%To enable other researchers to question our biases,
%we take
%care to make our case studies fully reproducible.
%Reproducibility is an important methodological principle in other
%disciplines since it allows a community to confirm, refute, or even
%improve prior results.  In our view, in the field of software
%engineering, there are all too few examples of reproduced, and
%extended, results\footnote{ Exception: see  the reports of the
%PROMISE workshop \url{http://promisedata.org/repository/papers.html}}.
%Accordingly,
%an appendix to this paper detailed how to install and run the
%software required to repeat the KC1\&ARES case studies.
%Such reproducibility allows other researchers to check if our advocacy
%of automated tools biases our conclusions.
%
%\subsection{Is There Really a Problem?}
%
To illustrate the local tuning problem,
consider
the COCOMO-81 development effort model~\cite{boehm81}.
This model assumes 
that the development $effort_i$ for $project_i$ is exponential on program size:
\begin{eqnarray}\label{eq:ab} 
effort_i &= &a \cdots KSLOC ^ b \cdots project_i\\
project_i &= &\prod^{15}_j EM_j
\end{eqnarray}
Here, $project_i$ is computed from 
the
15 {\em effort multipliers} $EM_i$ shown in \fig{cparems}.
{\em Local calibration} (LC)~\cite[p526-529]{boehm81} is a linear regression method 
that returns the
$a,b$ values that minimizes the difference between efforts predicted by \eq{ab} and the actual values seen in
$project_{1..i}$.

Based on historical data,
the COCOMO-81 model offered three sets of cliched $a,b$ values that varied
according to the different development {\em modes} of \fig{eso}.
Subsequent work~\cite{boehm00a}
in the late 1990s updated these values using 161 
projects and,  even after that update, the $a,b$ values 
were approximately equal to \fig{eso}.

%\begin{figure}
%\scriptsize
%\begin{center}
%  \begin{tabular}{rr|rr}
%       a & b      &  a & b\\\hline
%       3.72 &1.09 &  5.45 &1.05\\
%       4.06 &1.08 &  7.92 &0.85\\
%       4.22 &1.09 &  9.18 &0.85\\
%\end{tabular}
%\end{center}
%\caption{Local tunings on $<a,b>$.}\label{fig:tune}
%\end{figure}
Reusing calibrations from other projects
can lead to very inaccurate predictions.
Off-the-shelf ``untuned'' models have 
been up to 600\% inaccurate in their estimates, e.g. ~\cite[p165]{mukho92}~\cite{kemerer87}. 
For example, \fig{lc} shows 
local tuning experiments with some NASA COCOMO data\footnote{
\url{http://promisedata.org/repository/data/nasa93}}.
Local tuning was 
performed on  ten sub-domains of the NASA data (flight, ground, avionics monitoring, etc).
For each sub-domain, thirty times, 90\% of the data
was randomly selected for local tuning via LC.
The mean $<a,b>$ values are very different to the standard COCOMO-81 values of \fig{eso}.
That is, specialized organizations like NASA develop software that is very
different from the kinds of projects used to generate standard software process models.
For such organizations, models can't be reused verbatim from some prior setting.

Note the \hmm{underlined} entries
of \fig{lc}. These entries show that  
the standard deviation of the values learned from local tuning may be   very large.
That is, in these NASA COCOMO data sets,
there was insufficient data in these sub-domains to reach stable conclusions.
That is,
even if some local tuning data is available, it may still be incomplete.

Rather than rely on missing or incomplete
local data for model tuning, the rest of this paper explores how high-end
ASE tools
compare, relative to and manual methods, over the
the space of possible tunings within the COCOMO, COQUALMO, and THREAT
 software process models.
\begin{figure}[!t] 
{\scriptsize 
\begin{center}
\begin{tabular}{p{0.5in}|r|r|p{1.8in}} \noindent Mode & a & b & notes\\\hline
Organic & 3.2 & 1.05 & projects from relatively small software teams
develop in-house software in a highly familiar environment.\\\hline
Embedded & 2.8& 1.2 & projects operating within (is embedded in) a
strongly coupled complex of hardware, software, regulations, and
operational procedures.\\\hline Semi-Detached & 3.0 & 1.12 & An
intermediary mode between organic and embedded.  
\end{tabular}
\end{center}} 
\caption{COCOMO-81 development modes.}\label{fig:eso} 
\end{figure}
\begin{figure}[!t]
\scriptsize
\begin{center}
\begin{tabular}{|rr|rr|}
\multicolumn{2}{c}{a}
&\multicolumn{2}{c}{b}\\\hline
$\mu$&$\sigma$  & $\mu$ & $\sigma$ \\\hline
3.72& 0.51& 1.09& 0.037\\
4.06& 0.63& 1.07& 0.04\\
4.27& 0.78& 1.02& 0.05\\
5.06& \hmm{2.08} & 1.02& 0.09\\
5.74 & \hmm{2.13}& 1.00 &0.084\\
5.78& 1.5&0.85& 0.094\\
6.28& 0.46& 0.93& 0.02\\
6.35& 1.26& 0.97& 0.05\\
6.35& 1.49& 0.92& 0.05\\
9.18& \hmm{2.97}&0.88& 0.081\\\hline
\end{tabular}
\end{center}
\caption{30 repeats of local tuning on  90\% samples. }\label{fig:lc}
\end{figure}

\newpage
\input{models}

\section{Search Algorithms}
Having defined the internal space of our models, we now search it.
A simulated annealer is used to  sample that space and a back select
algorithm is used to  rank feature ranges according to how well they reduce effort/ defects/ 
threats.
\subsection{Simulated Annealing}
{\em Monte Carlo} algorithms randomly sample the space
of possible controllable model states. 
A  {\em Metropolis} Monte Carlo algorithm~\cite{metropolis53}
creates new states
by  small mutations to some
{\em current} state. 
If a new state is ``better'' (as assessed via a
fitness function also called the {\em energy function}),  it becomes the new {\em current}
state used  for future
mutations.  Otherwise, a Boltzmann acceptance
criteria is used to probabilisticly decide to
assess the new state: the worse the new state, the less
likely that it becomes the new current state.
The algorithm is silent on the mutation mechanism. For our experiments,
we freeze $\frac{2}{3}$ of the features and
randomly  select values for the rest.

In 1983, Kirkpatrick et.el.~\cite{kirkpatrick83} 
proposed a modification to the Metropolis algorithm inspired
by a procedure used to make the strongest possible glass. Initially,
the glass is heated to allow atoms to move freely. The temperature
is then slowly lowered such that the atoms can find the stablest
configuration with lowest energy. 
Hence, a {\em simulated annealing} (SA) algorithm adds
a ``temperature'' variable to the Boltzmann accept criteria
such that, at high temperatures, it is more likely that the
algorithm will jump from to a worst new current state.
This allows the algorithm to jump out of local minima while
sampling the space of options. As the temperature cools,
such jumps become less likely and the algorithm reverts
to a simple hill climber.

\begin{figure}[!t]
\scriptsize
\begin{verbatim}
function sa(kmax) 
  s := s0; e := E(s)           // Initial state, energy.
  sb := s; eb := e             // Initial "best" solution
  k := 0                       // Energy evaluation count.
  while k < kmax and e > emax  // Loop
    sn := neighbour(s)         //   Pick some neighbour.
    en := E(sn)                //   Compute its energy.
    if en < eb then            //   Is this a new best?
      sb := sn; eb := en       //     Yes, save it.
    if random() < P(e, en, temp(k/kmax)) 
     then s := sn; e := en     //     Maybe move to a worse state
    k := k + 1                 //   One more evaluation done
  return sb                    // Return best
\end{verbatim}
\caption{Mutate an initial state $s0$ to some $neighbour$ region.
A new solution $sn$ (with new energy $en$) is the best solution $sb$ 
if it has
less energy
$or$ if it is selected by
 probability function $P$.
}\label{fig:sa}
\end{figure}

According to the annealing metaphor, better solutions have lower ``energy''
so the fitness function $E$ is defined such that $E \ge 0$ and
{\em lower} values are
$better$. 
For our purposes, we model effort ($Ef$), defects ($De$) and threats ($Th$) 
in a 3-D space where the energy
is the  distance to the origin:
\[\small
E=\left(\sqrt{\overline{Ef}^2 + \overline{De}^2 + \overline{Th}^2}\right) / \sqrt{3}
\]
Here, $\overline{x}$ is a normalized value 
$0 \le \frac{x - min(x}{max(x) - min(x)} \le 1$. 
Hence, our energy ranges $0 \le E \le 1$ and
{\em lower} energies are {\em better} (less effort, defects, threats).

The acceptance criteria $P$ for a new state is defined using
the current temperature $T$, 
the energy of the current solution ($e$),
and the energy of the new mutation ($en$):
\[
P(e,en,T) = e^{(e -en)/T}
\]
$T$ is defined to $decrease$ as the simulator loops through $k= 1\ldots kmax$ iterations.
We use
$T = e^{-100*k/kmax}$.

\begin{figure}
\begin{center}
\includegraphics[width=2.5in]{jpl1.pdf}
%eps/JPL.pdf}
\end{center}
\caption{Processing a JPL requirements model. }\label{fig:jpl}
\end{figure}
Two advantages of SA algorithms are their {\em implementation simplicity} and their {\em ability to handle
non-linear models}:
\bi
\item {\em Simplicity:}
\fig{sa} illustrates the simplicity of the algorithm. Memory is only required
for one current solution ($s$), one new solution ($sn$) and one best solution ($sb$) that stores
the best solution seen at any time over the simulation.
\item
{\em Non-linear models:}
 Previously~\cite{fea02a,me03i}, we have applied SA to non-linear JPL requirements models
where minimizing the cost of project mitigations can decrease the number of requirements
achieved by that project. Hence, decreasing {\em both} the cost and achieved requirements
is a non-linear problem that must trade between minimizing cost and increasing
requirements coverage.
The top-left line  of \fig{jpl} divides the behavior of the JPL requirements models
before  and after  simulated annealing. 
As shown below the line, 
initial Monte Carlo sampling of the possible mitigations lead to a large range
of costs and benefits. Simulated annealing found a set of mitigations
that lead to the small cloud of solutions above the line. Compared to the initial samples, these
new solutions had decreased cost, increased benefit (number of requirements covered), and decreased variance (shrank
the space of solutions). 
\ei
Two disadvantages of SA algorithms are their {\em incompleteness} and the {\em
complexity}
of their solutions:
\bi
\item 
{\em Incompleteness:} In our domain, we have some evidence that the incomplete nature of the heuristic SA
search is not a major problem.
\fig{saeg} shows a sample run of our SA tool running on our process models
for $K=10,000$ simulations. 
As $k$ increases for $1{\ldots}10,000$, it becomes less and less
likely that a better $best$ has been missed. Hence, we run our simulations
for ten times the period it takes for $best$ to stabilize (at $k\approx 1000$). 
\begin{figure}
\begin{center}
\includegraphics[width=2.7in]{saeg1.pdf}
\end{center}
\caption{Dots and lines show
SA output from current and best solution (respectively)  after $k$ simulations.
}\label{fig:saeg}
\end{figure}
\item
{\em Complexity:}
Simulated annealers offer constraints to all controllable inputs. Often this
is an over-constrained solution. 
In many domains, a repeated empirical result is a {\em feature subset selection}
effect; i.e. many features can be pruned from a model without damaging
the predictive power of that model~\cite{kohavi97,hall03}.
For example, Kohavi~\cite{kohavi97}
present  studies where using just 19\% of the available features
changed prediction accuracy by just 2.14\% (on average).
For another example,
when feature subset selection was applied to the JPL requirements model of \fig{jpl}, we found that
up to $\frac{2}{3}$-rds of the features can be left unconstrained,
without effecting the conclusions~\cite{fea02a}. 
\ei
%
%\begin{figure}[!t]
%\scriptsize
%\begin{center}
%\begin{tabular}{|l|r|r|r|p{0.95cm}|}\cline{2-4}
%\multicolumn{1}{c}{}     & \multicolumn{3}{|c|}{av.  number of features} \\\hline    
%dataset         &before &after &$\frac{after}{before}\%$ &accuracy change\\\hline    
%breast cancer &10  &2.9 &29\% &+0.14\%   \\\hline    
%cleve           &13  &2.6 &2\%  &+5.89\%   \\\hline 
%   crx             &15  &2.9 &19\% &+4.49\%   \\\hline  
%  DNA             &180 &11  &6\% &+3.63\% \\\hline    
%horse-colic     &22  &2.8 &13\% &+1.63\%   \\\hline
% Pima            &8   &1   &13\% &+0.79\%   \\\hline   
% sick-euthyroid &25  &4   &16\% &+0.38\%   \\\hline    
%soybean         &35  &12.7 &36\% &+0.15\%  \\\hline\hline    
%average         &38.5&4.99 &19\% &+2.14\% \\\hline
%\end{tabular}
%\end{center}
%\caption{Feature subset selection results from~\cite{kohavi97}}\label{fig:fssr}
%\end{figure}

In terms of  the goal of this paper, exploring feature subset selection
is very important.  Before we can {\em remove} superfluous features,
we must first  {\em rank} them according to their effectiveness.
As we shall see, this ranking will be insightful to the task of
assessing the relative value of high-end ASE
tools and manual methods
for reducing effort/ defects/ threats.

\subsection{Support-Based Bayesian Ranking}

STAR 1.0 ranks the features ranges
 after $K$ runs of a simulated annealer. To compute this
ranking, STAR 1.0 divides all the $K$ solutions
into:
\bi
\item {\em Best:} those associated with the  BEST\% solutions
(i.e. those with the BEST\% {\em least} energy); 
\item And the {\em rest}.
\ei
It then computes the probability that a range is found in $best$.
using Bayes' Theorem.
Informally,
the
theorem says $next=old*new$ i.e. what we'll believe {\em next}
comes
from how {\em new} evidence effects {\em old} beliefs.  More
formally:\[P(H|E)=\frac{P(E|H) P(H)}{P(E)}\]i.e. using
evidence $E$ and a prior probability $P(H)$ for
hypothesis $H\in\{best, rest\}$.
the theorem calculate a  posteriori  probability
$P(H|E)$. 
Simple  Bayes classifiers are often called ``n\aive'' since they assume
independence of each feature. While this assumption
 simplifies the implementation  
(frequency
counts are required only for each feature),
 it is possible
that
correlated events are missed by this ``n\aive'' approach.
Domingos and Pazzani show theoretically
that the independence assumption is a problem in a vanishingly small percent
of cases~\cite{domingos97optimality}. This explains the repeated
empirical result that, on average, seemingly n{\aive} Bayes classifiers perform as well
as other seemingly more sophisticated schemes (e.g. see Table~1 in~\cite{domingos97optimality}).


When applying the theorem, it is standard practice~\cite{witten05} to compute  {\em likelihoods}
from observed
frequencies, then normalize these over the sum of all likelihoods to create
probabilities (this normalization cancels out $P(E)$, so we will not refer to it again).
For example, after $K=10,000$ runs, we might have 1,000 lowest 10\% {\em best} solutions and 9,000
{\em rest}. If the range $rely=vh$ appears 10 times in the $best$ solutions, but only
5 times in the $rest$, then:

{\scriptsize
\begin{eqnarray}\nonumber
E&=& (reply=vh)\\\nonumber
P(best)&=&1000/10000 = 0.1\\\nonumber
P(rest)&=&9000/10000 = 0.9\\\nonumber
freq(E|best)&=&10/1000 = 0.01\\\nonumber
freq(E|rest)&=&5/9000 = 0.00056\\\nonumber
like(best|E)&=&freq(E|best) \cdot P(best)=0.001\\\nonumber
like(rest|E)&=&freq(E|rest) \cdot P(rest)= 0.000504\\\label{eq:ten}
P(best|E)&=& \frac{like(best|E)}{like(best|E) + like(rest|E)}=0.66
\end{eqnarray}}

(Note that, in this two-class system, {\em increasing} the probability of $best$
also {\em decreases} 
the probability of $rest$.)

Empirically, we have found that \eq{ten}
is a poor ranking method for feature range $E$. The above example shows why.
The probability $P(best|E)$ is moderately large (0.66) but the {\em support}
for this range is very weak: \mbox{$freq(E|best) = 0.01$}. That is, \eq{ten} is a poor ranking method
since it can be distracted by low frequency evidence.

To avoid low frequency evident, we augment \eq{ten}
with a support term.
Such a support terms should $increase$ as the frequency of a range $increases$,  i.e. 
$like(x|best)$ is a candidate support measure.
In prior work~\cite{clarke05}, and in the results shown below,
we find that \eq{ps} is a very useful range ranging method: 
\begin{equation}\label{eq:ps}\small
P(best|E) * support(best|E) = \frac{like(x|best)^2}{like(x|best) + like(x|rest)}
\end{equation}
To apply \eq{ps}, STAR 1.0 runs in six phases.
In terms of standard machine 
learning theory, step 1 generates a training set; steps 2,3,4 do some generalization;
and step 5 tests the learned theory on data not seen during training.

\be
\item To sample the ranges from the models,
STAR 1.0 runs the simulated annealer  $K_1=10,000$ times to generate results like \fig{saeg}. 
Note that this sample will include the best solutions.
\item The data seen in the $K_1$ samples is then discretized into $D=10$ bins.
Discretization converts a continuous range into a histogram
with $n$ break points $b_1\ldots b_n$ where $\left(\forall i < j : b_i \le b_j\right)$. 
After discretization,
many observations can fall into the same range
between $b_i$ and $b_{i+1}$
at frequency counts $c_i$. This study used equal width discretization; i.e. 
\[\forall i,j:
\left(b_i - b_{i-1}\right) = 
\left(b_j - b_{j-1}\right)\]
\item
The ranges are then classified
into those seen in the top BEST\% {\em best} solutions and the (100-BEST)\% {\em rest};
\item
The ranges are then ranked in increasing order by their probability*support (\eq{ps}) of appearing 
in the X\% best outcomes;
\item
STAR 1.0 then runs $K_2$ experiments with the models where the top $N$ ranges are pre-set and the
remaining ranges can be selected at random;
In terms of the business case for automated software engineering tools,
this step is crucial since it would rank high-end ASE tools alongside a variety
of manual methods. 
\item
STAR 1.0 returns the $X$ value which report the least effort, defects, and threats.
\ee

To run our experiments, we had to apply our engineering judgment to set the parameters:
\[K_1=10,000, K_2=10,000, D=10, BEST=10\]Initially, we planned experiments that varied these parameters,
as well as trying other discretization policies.
However, our initial results were so promising (see below) that we are not currently motivated
to explore 
other parameter settings (such an exploration is planned for future work).

\begin{figure}[!t]
\begin{center}
\scriptsize
\begin{tabular}{c|lrr|lr}
       &\multicolumn{3}{c|}{ranges}      &\multicolumn{2}{c}{values}\\\hline
project&feature&low&high&feature&setting\\\hline
&prec&3&5&flex&3\\
ARES &pmat&4&5&resl&4\\
GNC&docu&3&4&team&3\\
&ltex&2&5&time&3\\
&sced&2&4&stor&3\\
&KSLOC&75&125&data&4\\
&&&&pvol&3\\
&&&&ruse&4\\
&&&&rely&5\\
&&&&acap&4\\
&&&&pcap&3\\
&&&&pcon&3\\
&&&&apex&4\\
&&&&plex&4\\
&&&&tool&5\\
&&&&cplx&4\\
&&&&site&6\\\hline
&prec&1&2&data&3\\
&flex&2&5&pvol&2\\
Orbital&resl&1&3&rely&5\\
space&team&2&3&pcap&3\\
plane&pmat&1&4&plex&3\\
&stor&3&5&site&3\\
&ruse&2&4&&\\
&docu&2&4&&\\
&acap&2&3&&\\
&pcon&2&3&&\\
&apex&2&3&&\\
&ltex&2&4&&\\
&tool&2&3&&\\
&sced&1&3&&\\
&cplx&5&6\\
&KSLOC&75&125
\end{tabular}
\end{center}
\caption{Two case studies.}\label{fig:cases}
\end{figure}

\section{Case studies}

STAR 1.0 was applied to the two case studies of
\fig{cases}:
\bi
\item ARES GNC: the guidance and navigation control system of NASA's ARES rocket.
\item OSP: The Orbital Space Plane prototype (a system explored by NASA in the  1990s)
\ei
In \fig{cases},  {\em values} are {\em fixed} while {\em ranges} represent a space of options.
In the sequel, this observation will be important:
\bi
\item
ARES
constrains most of its features to fixed values.
\item 
OSP allows more variation in feature ranges.
\ei

$K_1$ runs of the simulated annealing were then conducted.
Feature ranges not mentioned in \fig{cases} were selected at random from \fig{lc} and \tion{models}.
The {\em controllable} project feature ranges were sorted
according to \eq{ps}:
\bi
\item
 By {\em controllable}, we mean the kinds of changes
a project manager could make to her project. 
\item
{\em Uncontrollable} features, on the other hand,
are features like the correlation between, say, analyst capability and development effort. 
\ei
For this study, the controllable features are listed in \fig{cparems} while the uncontrollable
features 
are the internal model parameters of \fig{lc} and \tion{models}. 

\newcommand{\hey}{~~~$\lhd\lhd$}
\begin{figure*}[!t]
\begin{center}
\begin{tabular}{c|c}
ARES GNC & OSP: Orbital Space Plane\\\hline
& \\
\begin{minipage}{.4\linewidth}
{\tiny
\begin{tabular}{r|r@{~=~}l}
X& feature & range\\\hline
1&docu&3\\
2&pmat&5\\
3&ltex&5\\
4&pmat&4\\
5&sced&3\\
6&prec&5\\
7&docu&4\\
8&ltex&4\\
9&prec&4\\
10&sced&2\\
11&automated analysis&6\hey\\
12&sced&4\\
13&prec&3\\
14&execution testing and tools&5\hey\\
15&execution testing and tools&6\hey\\
16&peer reviews&4\\
17&automated analysis&4\\
18&peer reviews&5\\
19&execution testing  and tools&4\\
20&peer reviews&2\\
21&automated analysis&5\hey\\
22&peer reviews&6\\
23&peer reviews&3\\
24&execution testing and tools&2\\
25&ltex&3\\
26&automated analysis&3\\
27&automated analysis&2\\
28&execution testing and tools&1\\
29&execution testing and tools&3\\
30&peer reviews&1\\
31&automated analysis&1\\
32&ltex&2
\end{tabular}
~\\
~\\
~\\
~\\
~\\
~\\
}
\end{minipage}
&
~~~~~~~~~~\begin{minipage}{.4\linewidth}
{\tiny
\begin{tabular}{r|r@{~=~}l}
X& feature & range\\\hline
1&acap&3\\
2&cplx&5\\
3&tool&3\\
4&apex&3\\
5&pcon&3\\
6&ruse&2\\
7&time&3\\
8&docu&2\\
9&ltex&4\\
10&prec&2\\
11&team&3\\
12&stor&3\\
13&resl&3\\
14&sced&3\\
15&pmat&4\\
16&team&2\\
17&prec&1\\
18&flex&5\\
19&sced&2\\
20&pmat&3\\
21&pcon&2\\
22&stor&4\\
23&flex&4\\
24&apex&2\\
25&docu&3\\
26&ltex&3\\
27&resl&2\\
28&tool&2\\
29&time&4\\
30&ruse&3\\
31&execution testing and tools&6\hey\\
32&peer reviews&5\\
33&resl&1\\
34&cplx&6\\
35&peer reviews&6\\
36&automated analysis&2\\
37&execution testing and tools&5\hey\\
38&acap&2\\
39&pmat&2\\
40&automated analysis&5\hey\\
41&automated analysis&6\hey\\
42-62& \multicolumn{2}{c}{not shown (for space reasons)}\\
\end{tabular}
~\\
}
\end{minipage}
 \\
%\includegraphics[width=3.2in]{data/ares/Effect_ares.pdf} &
%\includegraphics[width=3.2in]{data/ares/Effect_kc1.pdf}
%\\
 \includegraphics[width=3.2in]{data/ares/Defects_ares.pdf} &
\includegraphics[width=3.2in]{data/ares/Defects_kc1.pdf}
\\
\includegraphics[width=3.2in]{data/ares/Effort_ares.pdf} &
\includegraphics[width=3.2in]{data/ares/Effort_kc1.pdf}
\\
\includegraphics[width=3.2in]{data/ares/Threats_ares.pdf}  &
\includegraphics[width=3.2in]{data/ares/Threats_kc1.pdf}\\
 Number of constraints &
Number of constraints \\
\end{tabular}
\end{center}
\caption{Results from  {\em range ranking} and {\em back select}, 
shown {\em top} and {\em bottom} (respectively). }\label{fig:backwards}
\end{figure*}
The results from STAR 1.0's phase 4 (range ranking) and phase 5 (back select)
are shown in \fig{backwards}. The list at the top of
that figure sorts the ranges of the controllable  features according
to  \eq{ps}; best to worst, top to to bottom. The plots at the
bottom of that figure show the effect on defects/ effort/ threats
of imposing 
ranges 1..X, then selecting the remaining ranges at random. 

When imposing  constraints 1..X, sometimes one feature will be mentioned
more than once. For example, for ARES, the {\em policy} 
X=1..4
includes one value for $docu$, one for $ltex$, and two values for $pmat$.
When exploring  the  constraint  X=1..4, we hence sample from
the set:
\[
ltex\in\{3\} \wedge ltex\in\{5\} \wedge pmat\in\{4,5\}
\]

In \fig{backwards},
the {\em median} curves shows the 50\% percentile and {\em spread}
curves shows
the difference between the 75\% percentile and the 50\% percentile.
In those curves, as $X$ increases (working left to right), more
constraints are imposed on the simulation. 
The  spreads in these curves
arise from the effects of the unconstrained features.
Even with constraints set to all the controllable features,
(at X=32 for ARES, and X=62 for OSP), some spread remains. This
spread arises from STAR 1.0's sampling of the space internal
to our process models.

The ARES effort and threats curve were {\em non-responsive};
i.e. median outputs
were not effected by adding constraints:
\fig{cases} explains this unresponsiveness: ARES constrains more
features. Hence, at least for ARES effort and threats, STAR 1.0 can not change
median values. 

On the other hand, the ARES defect curve and all the OSP curves were 
{\em responsive}, i.e. adding constraints
changed defects/ effort/ threats.
For such responsive curves, 
STAR 1.0's {\em policies} can have a dramatic effect; e.g.
STAR 1.0 found ranges that decrease OSP's median effort by over 300\%.

One of our pre-experimental concerns was that no stable
{\em polices} could be drawn from STAR 1.0's widespread sampling of the space
internal to our process models. At least for the case studies
explored here, this concern was unfounded:
\bi
\item
In the case of the responsive curves, the change effected by {\em policies} 
1..X
occurred {\em despite} wide-spread variation to the internals
of the process models. 
\item
In the case of the non-responsive curves,
we now have greater confidence in them since
they are stable across a wide range of project settings and internal variations to
the process models. For example, according to this study, the effort for ARES 
has 50\% to 75\% range of 600 to 900 person months (median
plus spread).
\ei

The 
 ``$\lhd$'' 
symbol in \fig{backwards}
indicates where  high-end ASE tools appear in the rankings (i.e.
automated analysis or execution-based testing tools with COCOMO
ratings of 5 or 6):
\bi
\item
With ARES, high-end ASE tools are found at X=11,14,15,21.
\item
With OSP, those tools are found at 
X=31,37,40,41.
\ei
As mentioned above, effort and threats in ARES are non-responsive to STAR 1.0
so we need only study
the effects of applying 1..X on the other four curves.

For ARES, median defects minimizes at X=17;  i.e. high-end execution test
and tools  
plus very high-end automated analysis (range=6 in \fig{cparems}) are found
in the best {\em policy}  of X=1..17.
However, anything less than very high-end automated analysis
is not indicated; i.e. $automated analysis > 5$ was useful, but anything
less was not.

For Orbital Space plane, minimum median effort and threats occur around
X=15. This point corresponds to local minimum for defects that is 30\% 
larger than the global defect minimum at
X=32.  Note that, at X=32, the development effort is now  nearly
doubled from its minimum value at X=15. 
Hence, the best {\em policy} for  OSP
lies between X=1..15
(if project funds are limited) to X=1..32 (if
a 30\% decrease in defects is cost-justified). 
In either case, high-end automated analysis tools
are not indicated
(since these don't appear till X=40 and X=41). As to execution 
testing and tools:
\bi
\item
For cost-adverse
projects, that will accept the local defects minimum at X=15, high-end
execution testing and tools
not indicated. 
\item
Very high-end execution testing and tools
(range=6 in \fig{cparems}) 
is only indicated 
for risk-adverse projects that
are willing to double the development effort to achieve a minimum level of
defects.
\ei

\section{External Validity}\label{sec:ev}

The utility of this style of analysis is its external validity.
Conclusions reached from process models tuned to local data are biased
by that tuning. The conclusions reached here, on the other hand, are stable
over a large space of possible tuning biases.

The above conclusions are drawn from the internal space
of some software process models. Clearly, if
those process models are wrong,
then these conclusions are also wrong,
One mitigation for this risk is to
use the best available models.
COCOMO, COQUALMO, and THREAT 
have been extensively validated:
\bi
\item
Chulani et.al.~\cite{chulani99} reports a one study with a Bayesian
tuning algorithm using 161 projects. After tuning, a cross-validation
study showed that COCOMO produced effort estimates that are 
within 30\% of the actuals,
69\% of the time.
\item
Studies with the COCOMO-81 project database have shown that
the THREAT index correlates with well
with the $\frac{months}{KDSI}$ (KDSI= thousand
of delivered source lines of code).
This result is consistent with the
the base premise of THREAT; i.e.  bad management can delay
software delivery.
\item
COQUALMO was developed using extensive feedback
from the  COCOMO affiliates
group. This group comprises 
dozens
of companies, that have donated 161 data sets, that meet each year to discuss
improvements to the current set of
models\footnote{e.g. \url{http://sunset.usc.edu/events/2006/CIIForum/}}.
\ei

A more serious threat to our external validity is STAR 1.0's search bias.
There are $2^N$
possible combinations
of $N$ ranges
but STAR 1.0 only explores $N$ of them  during its back select
phase.
Hence, in theory, the rankings of \fig{backwards} may not accurately reflect 
the true relative effectiveness of the ranges.

There is some theoretical and empirical evidence that 
\eq{ps} offers an appropriate ordering.
Clarke has conducted a  theoretical study concluding that
\eq{ps} 
offers the wrong rankings in a very small 
percent of cases~\cite{clark05}.
This theoretical result also explains two empirical results:
\bi
\item
Clarke ran numerous experiments comparing
(a)~the estimated ``best'' combinations
of ranges
found by \eq{ps} and the (b)~actual
best combinations found by generating 
combinations,  then trying each one against the 
training data.  In all those experiments,
Clarke only found one minor case
where $ranking(a) \not= ranking(b)$.
\item
If \eq{ps}'s ordering was perfect,
then we would expect poor performance when
too few constraints were applied (important ideas ignored)
and when the solution was over-constrained
with too many constraints (too many superfluous details).
In the middle of these two extremes would be a low energy valley where
the constraints are most effective.
Note that such a U-shape is seen in 
all the responsive curves of \fig{backwards}.
\ei

\section{Related Work}

This paper is the first to:
\bi
\item
Incorporate a defect introduction/removal model
with an effort and threat predictor;
\item
Conduct that exploration with the STAR 1.0 simulated annealer, plus a Bayesian
ranking method.
\item
Explore the stability of the conclusions within widespread perturbation
of the internal model parameters
To the best of our knowledge, this paper is the most extensive
study to date of the effects of internal model instability
on the stability of process model conclusions.
\ei

%This paper was motivated by a lack of results regarding the relative
%merits of different methods.  To be fair, apart from \fig{coceffects},
%the general software engineering literature is weak on the relative
%merits of {\em any} set of methods.  We recently reviewed the
%(approx) 100 V\&V methods offered by the SE literature\footnote{
%Wallace and Fujii's classic definitions of V\&V~\cite{wallace89};
%the IEEE 2004 standard on V\&V~\cite{1012}; NASA's recommended V\&V
%practices~\cite{jackson97}}. While some evidence was offered for
%the value of general techniques\footnote{ E.g. improving software
%maturity decreases rework~\cite{king02}} there was very little
%precise evidence on the merits of a particular tool. Further, there
%was almost no comparisons of the relative effectiveness of pairs
%of methods applied to the same task.
%
%Lacking results from other fields, this study turned to model-based
%methods to assess alternate methods.

%Other researchers have explored meta-heuristic search methods
%(simulated anneling, genetic algorithms, fuzzy logic) for software
%engineering~\cite{seminal01,clark03,harman04}.
%In 2004, Rela~\cite{rela04} reviewed 123 publications in the field
%of search-based software engineering and reported that 44\% were
%related tot esting, while only 7\% of them related to project
%planning. None of that 7\% combined effort and defect and threat
%models or explored the effects of their conclusions on internal
%perturbations to their models.
%
Prior publications by the authors
have explored similar issues to this paper
using:
\bi
\item
The TAR1  minimal contrast set learner; and
a Monte Carlo
simulations of the COCOMO and THREAT models 
(with fixed internal parameters)~\cite{me00e}.
\item
Later versions of TAR (TAR2, TAR3)
on different case studies and different models (again,
with no perturbation of internal model details~\cite{raffo05b,raffo05c,me02f}.
\ei
A major advantage of STAR 1.0 over TAR
is performance speed. Using STAR 1.0, \fig{backwards} takes
less than a minute to generate. A similar range ranking,
generated
by TAR1,
would have
required an overnight run~\cite{me00e}. TAR2 and TAR3
were much faster but even those optimized systems would
require
20 to 30 minutes to 
generate feature range rankings.

This work used COCOMO-family software process
models and
numerous alternate models have been developed by the software process
modeling community\footnote{e.g.
\url{http://www.icsp-conferences.org/icsp2007/}}. Some of these are
specialized models built to serve the needs of a particular software
development organization. The COCOMO models, on the other hand,
were built to generalize over the community of the COCOMO affiliates.

We have recently become aware that
STAR 1.0's Bayesian ranking methods
is analogous to Ruinstein's cross-entropy method
for controlling mutation during simulated annealing~\cite{rubinstein04}.
Cross-entropy requires some guesstimate of the probability
distribution of the feature ranges (e.g. Gaussian). It is 
applied during simulated annealing to speed up convergence
to best solutions (by avoiding rare events). 
STAR 1.0's ranking method, on
the other hand, is non-parametric relying as it does on a pre-discretized
space. It is also applied as a post-processor to the SA. In future
work, we will explore more  the connection of Bayesian ranking to
cross-entropy.


\section{Conclusion}

Tim Lister argued at ICSE 2007 that, in terms of successfully
completing a project, ``sociology beats technology''. In this
view, the best way to ensure the successful completion of a project
is to focus on human factors such as the capability of the analysts,
and not the tools they use.

To check the Lister hypothesis, we conducted a search over the space
of possible tunings within the COCOMO, COQUALMO, and THREAT models.
At issue was whether or not we could find high-end ASE tools amongst
the minimum set of project settings most reduced
defects/ threats/ and effort.  

To define ``high-end'', we used the
COCOMO ontology of \fig{cparems}. COCOMO scores its features 1 to
6 and we sought values of 5 or 6 for automated analysis or
execution-based testing tools amongst the minimum set of most effective
settings.

Standard practice is to use local tuning data to adjust models
to COCOMO, COQUALMO and THREAT to local conditions.
However, as discussed in \tion{inc},
we have often found that such local tuning data is hard to obtain,
or is incomplete. Hence, in
this study, we used   simulated annealing to search 
the space of possible tunings. 

The ranges seen in that simulated
annealing were then ranked using a novel support-based Bayesian ranking
method.
This ranking was then searched, best to worst, and experiments
were performed on COCOMO, COQUALMO, and THREAT using just the top X
ranged ranges. In those experiments,
the remaining ranges were selected at random from either the
ranges known for a particular project (see \fig{cases}) or
the range of internal tunings (see \tion{models}).

Our tool, called STAR 1.0, returns the feature ranges
that 
appear in the top 1..X
ranked ranges
required 
to achieve some minimum effort/ defect/ threat value.  
In two case studies with NASA systems (OSP and ARES),
high-end ASE tools were found in the top 1..X
ranges.

Hence, we find that
Lister is both right and wrong:
\bi
\item Human factors
such personnel experience and capability are vital to successfully completing
a project; for example, in \fig{backwards},
sociological factors such as
personnel continuity
appears at  
X=5 of OSP.
 \item
 However, we find 
in our two case studies that 
it is also vital
to augment these
human factors 
with high-end
ASE tools.
\ei
 

Our conclusions are project-specific but the method of generating
them is general to any project that can be described in terms of
the \fig{cparems} table. An appendix of this paper describes how
to download, install, and use STAR 1.0 to find the most important
feature ranges for some project other than OSP and ARES.

\bibliographystyle{abbrv}
{\scriptsize
\bibliography{refs}}  % sigproc.bib is the name of the Bibliography in this case
\appendix
\section{Obtaining and Using STAR 1.0}

\begin{verbatim}
wget http://unbox.org/wisp/tags/STAR/1.0/STAR_1.0.zip
unzip STAR_1.0.zip
make  # requires gcc
cd ~/STAR
cd eg
./1  > ares.out
./2  > osp.out
\end{verbatim}

To run STAR 1.0 on other projects, copy and edit
one of the projects files
in \verb+~/STAR/STAR_projects+. Note that you will have to
edit {\em both} a x.ranges and x.values file.
Then copy and edit (e.g.) \verb+~/STAR/eg/1+ to point to
your edited project details.

\end{document}


