\documentclass[10pt,twocolumn]{article}
todo citations, figures, better text from real proposal
usepackage{../../tex/latex8}
\journalname{27th  IEEE/NASA Software Engineering Workshop, Dec
4-6, 2002 (submitted)\\}
date{WP ref: 02/superodc/superodc. \today}
usepackage{graphicx,cite,url,alltt,times}
usepackage{algorithmic}
usepackage{../../tex/bar}
\usepackage{../../tex/lineno}
\topmargin -2cm
\newcommand{\epsorpdf}{pdf}
newcommand{\epsorpdf}{eps}
newcommand{\bd}{\begin{definition}}
newcommand{\ed}{\end{definition}}
XXX union of the cardinaility worng at 1262
 1,2,4-
 3,5
newenvironment{smallitem}
{\setlength{\topsep}{0pt}
 \setlength{\partopsep}{0pt}
 \setlength{\parskip}{0pt}
 \begin{itemize}
 \setlength{\leftmargin}{.2in}
 \setlength{\parsep}{0pt}
 \setlength{\parskip}{0pt}
 \setlength{\itemsep}{0pt}}
{\end{itemize}}
newenvironment{smallenum}
{\setlength{\topsep}{0pt}
 \setlength{\partopsep}{0pt}
 \setlength{\parskip}{0pt}
 \begin{enumerate}
 \setlength{\leftmargin}{.2in}
 \setlength{\parsep}{0pt}
 \setlength{\parskip}{0pt}
 \setlength{\itemsep}{0pt}}
{\end{enumerate}}
pagestyle{plain}
title{Improved software engineering decision support
hrough automatic argument reduction tools}
\small \author{Tim Menzies \\
 West Virginia University\\
 \protect\url{tim@menzies.us}
 \and
 James Kiper, \\Miami University, \\\protect{\url{kiperjd@muohio.edu}}\\
and Martin S. Feather, \\Jet Propulsion Laboratory, \\California
nstitute of Technology,
\\protect\url{martin.s.feather@jpl.nasa.gov} }}
begin{document}
\linenumbers\modulolinenumbers[5]
\maketitle \thispagestyle{empty}
\begin{abstract}
\input{abstract}
\end{abstract}
newcommand{\fig}[1]{Figure~\ref{fig:#1}}
newcommand{\bfig}[1]{(\fig{#1})}
newcommand{\tbl}[1]{{\fig #1}}
newcommand{\tion}[1]{\S\ref{sec:#1}}
newcommand{\btion}[1]{(\tion{#1})}
newcommand{\stion}[1]{\mbox{(see \tion{#1})}}
newcommand{\bi}{\begin{smallitem}}
newcommand{\be}{\begin{smallenum}}
newcommand{\ei}{\end{smallitem}}
newcommand{\ee}{\end{smallenum}}
begin{abstract}
ome of the most influential decisions about a software system are made
n the early phases of the
oftware development life cycle.  Those decisions about requirements and
esign are generally made by
eams of software engineers and domain experts who must weigh the complex interactions among
equirements and the associated developmental and operational risks of those requirements.
ome of these early life cycle decisions are more influential, or perhaps fateful, to subsequent
oftware design and development than are others.
hen debating about complex systems with a large number of options,
umans can often be slower than an AI system at identifying the {\em
lusters of key decisions} that give the most leverage. By focusing a
roup of human domain experts or software engineers on these key decision clusters,
ore time can be devoted to these pivotal
ecisions and less time is wasted on irrelevancies.
end{abstract}
footnotetext{SEDECS’2003: the 2nd International Workshop on Software
ngineering Decision Support (part of SEKE2003); June 20
003~\url{http://www.ksi.edu/seke/seke03.html}.Date \today. WP ref:
3/sekew/star1.
RL:\protect\url{http://tim.menzies.us/pdf/03star1.pdf} }
section{Introduction}
n early phases of a system development, many crucial decisions about software
re taken.  In the process of requirements elicitation and analysis, it is common
or clients to request much more than can be afforded - either because of
udgetary constraints, time limitations, or other issues like safety concerns.
hat is, there are risks associated with requirements.  (In this context, we take
 broad definition of risk - anything that gets in the way of satisfying a
equirement.)
here are often things that can be done to lower the risk, and thus increase the
hances of meeting requirements.  In fact any verification or validation activity
an be considered to be a risk reducer.  In a typical system development, it is
ommon for the number of requirements and associated risks, and therefore the
umber of possible mitigating activities to be quite large.  Each of these mitigating actions has a cost associated with it.  The Defect Detection and
revention (DDP) tool provides an ontology for representing these requirements, risks,
nd mitigations, and for reasoning about them.  DDP is a decision support tool
o be used in the early part of the life cycle of system development.  In this
aper, we describe several ways that we are automating this decision support.
footnotetext{ This work was sponsored by the NASA Office of Safety
nd Mission Assurance under the Software Assurance Research Program
ed by the NASA IV\&V Facility and conducted at the University of
iami, West Virginia University (partially supported by NASA contract
CC2-0979/NCC5-685) and at the Jet Propulsion Laboratory, California
nstitute of Technology (under a contract with the National
eronautics and Space Administration). The JPL work was funded by
ASA's Office of Safety and Mission Assurance, Center Initiative UPN
23-08.   That activity is managed locally at JPL through the
ssurance and Technology Program Office. The second author's research
s also supported in part by National Science Foundation Grants
CR-0204139 and CCR-0205588. Reference herein to any specific
ommercial product, process, or service by trade name, trademark,
anufacturer, or otherwise, does not constitute or imply its
ndorsement by the United States Government or the Jet Propulsion
aboratory, California Institute of Technology.
}
subsection{Imagine the Scene ...}
A team of NASA's top experts are debating
ptions on some complex deep space mission. The mission is in its
arly planning stages so few of the details are fixed. The science
eam wants to add a new instrument package to the mission. But the
ropulsion experts are already worried about the payload mass and any
ddition worries them. Also, the electronics team members are worried
bout the added stress to power consumption and heat production on the
n-board systems.  A spirited
iscussion follows in which each team tries to explain the costs and
enefits of their various proposals.
n the midst of this heated debated, a screen flickers. The AI system monitoring the debate
roduces a visual display of the several clusters of alternate decisions, each of which meets
ost restrictions and satisfies requirements to an acceptable level.
he visualization of these clusters helps the team to realize that some sets of decisions, although feasible, do not fit their development organization.  Simultaneously, the automated monitoring system
as just realized that of the dozens issues remaining, a resolution on (e.g.)
our matters makes debates about most of the other issues redundant.
he AI system presents to the team a decision cluster- a set of useful alternative decisions.  The team finds that it has consensus on two of those decisions, so those are quickly adopted. These two decisions greatly reduce the space of remaining discussions
nd the group finishes their debates in time
or lunch.
begin{figure}
includegraphics[width=3.2in]{sim.eps}
end{figure}
section{The DDP Tool}
The above scene is not science fiction- some of the
echnology has already been developed and applied to Jet Propulsion Laboratory
JPL) missions. At
PL, the DDP tool~\cite{corn01} is in use to organize interactive
nowledge acquisition and decision making sessions with spacecraft
xperts . The DDP tool and process works as follows:
begin{figure}
\hrule~
DDP assertions are either:
bi \item {\em Requirements}  (free text) describing the objectives
nd constraints of the mission and its development process; \item
\em Weights} (numbers) associated with requirements, reflecting
heir relative importance; \item {\em Risks} (free text) describing
vents that can damage requirements; \item {\em Mitigations}: (free
ext) describing actions that can reduce risks; \item {\em Costs}:
numbers) effort associated with mitigations, and repair costs for
orrecting Risks detected by Mitigations; \item
{\em Mappings}: directed edges between
equirements, mitigations, and risks. \item
{\em Part-of relations}
tructure the collections of requirements, risks and mitigations; \ei
\hrule~
caption{DDP's ontology}\label{fig:ddpont}
end{figure}
bi
\item  6 to 20 experts are gathered together for short, intensive knowledge
cquisition sessions (typically, 3 to 4 half-day sessions).  These
essions {\em must} be short since it is hard to gather together
hese experts for more than a very short period of time. \item  The
DP tool supports a graphical interface for the rapid entry of the
ssertions. Such rapid entry is essential, lest using the tool slows
p the debate. \item Assertions from the experts are expressed in
sing an ultra-lightweight decision ontology (e.g. see~\fig{ddpont}).
he ontology {\em must} be ultra-lightweight since: \bi \item Only
rief assertions can be collected in short knowledge acquisition
essions. \item   If the assertions get more elaborate, then experts
ay be unable to understand technical arguments from outside their
wn field of expertise. \ei\ei
begin{figure}[!t]
begin{center}
includegraphics[width=3.5in]{arrt.eps}
end{center}
\small \begin{description} \item[Faces] denote requirements;
item[Toolboxes] denote mitigations; \item[Skulls] denote risks;
item[Conjunctions] are marked with one arc; e.g. {\em require1} if
\em require2} and {\em require3}. \item[Disjunctions] are marked
ith two arcs; e.g. {\em fault1} if {\em fault2} or {\em fault3}.
item[Numbers] denote weights; e.g. {\em action5}
reduces the contribution of {\em fault3} to {\em fault1}, {\em
ault1} reduces the impact of {\em require5}, and {\em action1}
educes the negative impact of {\em fault1}. \item[Oval]   denotes
tructures that are expressible in the latest version of the JPL
emantic net editor (under construction).
end{description}}
 \caption{A
  semantic net of the type used at JPL~\cite{fea01}. }\label{fig:arrt}
end{figure}
he result of these sessions is a network of influences connecting
roject requirements to risks to possible mitigations. A (highly)
tylized version of that network is shown in \fig{arrt}.
The ontology
f~\fig{ddpont} may appear too weak for useful reasoning. However, in
epeated sessions with DDP, it has been seen that the ontology is
ich enough to structure and simplify debates between NASA experts.
or example, DDP has been applied to over a dozen applications to
tudy advanced technologies such as \bi\item a computer memory
evice; \item gyroscope design; \item software code generation; \item
 low temperature experiment's apparatus; \item an imaging device;
item circuit board like fabrication; \item micro electromechanical
evices; \item a sun sensor; \item a motor controller;\item
hotonics; and \item interferometry. \ei In those studies,  DDP
essions has found cost savings exceeding \$1 million in at least two
f these studies, and lesser amounts (exceeding \$100,000) in the
ther studies. The DDP meetings have also generated numerous design
mprovements such as savings power or mass and shifting of risks from
ncertain architectural to better understood design. Further, at
hese meetings, some non-obvious significant risks have been
dentified and mitigated. Lastly, DDP can be used to document
esolutions to those debates. Hence, DDP is in use at JPL:
bi \item  not only as a group decision support tool (as it was
esigned to do); \item … but also a design rationale tool to document
ecisions. \ei
hat is not to say DDP cannot be improved.
begin{figure}
begin{center}
includegraphics[width=3in]{annealling.eps}
end{center}
caption{Simulated annealing, an example.}\label{fig:anneal}
end{figure}
subsection{Improving DDP with Simulated Annealing}
ptimizing risk mitigations means {\em minimizing} costs
hile {\em maximizing} benefits.  That is, it is a classic {\em
ptimization problem}. A commonly-used search technique for such
ptimization is {\em simulated annealing}~\cite{kirkpatrick83},
llustrated in \fig{anneal}. Simulated annealing is a kind of
ill-climbing search for finding a good solution. A simple
ill-climber simply jumps to the next best solution and can hence
iss  globally optimal solutions since it can't move to a near-by
igher peak if, to do so, means traveling down-hill across a valley.
imulated annealing avoids this problem using a ``jump'' factor that
s a function of a ``temperature'' variable. At high
`temperatures'', simulated annealing can sample more of the local
errain since it can jump up-hill {\em or} down-hill. As the search
roceeds and the ``temperature'' cools, simulated annealing jumps
ess and less. Eventually, the jumping mechanism ``freezes'' and
imulated annealing completes its search like a simple hill climber.
 simulated annealing capability is now part of the DDP
ool~\cite{corn02}.  This gives automated support for the difficult
hoices that project managers have to make among possible mitigations
eeded to reduce risk to acceptable levels.  It is common in systems being
eveloped at JPL for expert to identify 50 to 100 risks to requirements, and 50 to 100 mitigating actions that can be taken to reduce these risks to acceptable
evels.  Making a binary choice for each of 50 mitigations gives a search space of size $2^{50}$.  A simulated annealing algorithm can search this space in a few minutes to give an adequate solution, or can be run for several hours to give
 near optimal solution.
n a typical JPL system development, such a search could be conducted overnight,
etween DDP sessions.  This would provide the session participants the additional
nformation that they need to finish the requirements and/or high level design.
subsection{Improving DDP with Clustering}
hen search a complex space of decision options, simulated annealing
roduces a {\em near-optimal} solution.  That is, after a large number of steps
f the simulated annealing algorithm, a set of decisions is discovered that
eets the objective function, and meets it more closely than any other
ecision set checked.    However,
t should be noted that the data of this search space is generally produced
y expert judgment.  Although such judgments are typically quite good (and
re the best measure available), they cannot be assumed to be completely
ccurate.  Thus, other sets of decisions, that are judged to meet the objective
unction at a slightly lower level, may have other properties that make them more desirable.
begin{figure}
begin{center}
\includegraphics[width=1.7in]{clusters.eps}
end{center}
caption{Clusters of mitigations.}\label{fig:clusters}
end{figure}
or example, if there are 100 decisions to be made (perhaps 100 mitigating
ctivities that
he project manager may do to assure product quality), there are $2^{100}$
ossible  sets of decision that could be taken.  Of these,
t would not be surprising for
he simulated annealing to identify 1000 that are within 2\% of the optimal
alue of the objective function.  To ask a project manager or management team to
hose among these 1000 may be a daunting task.  We have been experimenting with
arious clustering strategies to reduce the complexity of this decision.  As seen in figure \fig{clusters}, this large number of decisions may be collected into a much smaller
f clusters.  Each cluster has the property that all its members share 95\% of the
ecision.  With this visualization, the human experts can now choose the cluster
hat best fits the characteristics of their development organization.  That is, if
he management team is aware that their team is especially strong in formal
pecification, but much weaker in model checking, then they would choose a cluster
hat includes more formal specification activities rather than model checking
ctivities.
begin{figure}[!t]
\footnotesize
begin{center}
includegraphics[width=1.7in]{jplb4.eps}
end{center}~\\
\bf \fig{tar2eg}.A: Before}. Here, one dot is one project plan; i.e.
ne possible setting to the 99 risk mitigation options.
\\
begin{center}\includegraphics[width=1.7in]{jplafter.eps}\end{center}~\\
\bf \fig{tar2eg}.B: After}. Results from applying
the constraints learnt by the TAR2 contrast
set learner.}
caption{An application of TAR2.
-axis=
`cost'' = sum of the cost of selected risk mitigations
lower is better).
-axis=
`benefit''= requirements coverage, less the effects of risk
more is better)}\label{fig:tar2eg}
end{figure}
section{Improving DDP with the TAR2 Tool}
In a typical use of DDP, experts sketch out mappings between requirements,
isks, and mitigations then search for the cheapest mitigations that
ost reduce risks. As discussed previously, this search can be overwhelming large. \fig{tar2eg}.A shows the results of 50,000 runs with DDP for one deep space mission with 99 possible
itigations; i.e. $2^{99}\approx10^{30}$ possibilities. This space is
oo large to explore thoroughly. In each run, a random set of mitigations were
elected each time.  Note the huge range of possible costs and
enefits.
10^{30}$  seems a dauntingly
arge number of options to explore. Fortunately, our TAR2 {\em contrast set
earner}~\cite{ying02,me02a,me02f,me02c,me01f,me01h} has shown
hat a heated discussion on most of the risk mitigations would be a
\em complete waste of time}. A contrast set learner finds the
ifferences in variable settings seen in different situations. For
xample, an analyst could ask a contrast set leaner ``what are the
ifferences between people with Ph.D. and bachelor degrees?''. TAR2
iffers from other contrast set learners such as TARZAN~\cite{me00e}
nd STUCCO~\cite{bay99} in that it searches for the {\em smallest}
ontrast set that {\em most} separates preferred and undesired
ehavior.
begin{figure}[!t]
\footnotesize
{\em
begin{center}
begin{tabular}{rcccl}
utlook & temp($^o$F) &  humidity & windy? & class
\\hline
unny&  85&     86&     false&  none\\
unny&     80&     90&     true&   none\\
unny&     72&     95&     false&  none\\
ain&  65&     70&     true&         none\\
ain&        71&     96&     true&   none\\\hline
     rain&         70&     96&     false&      some\\
     rain&      68&     80&     false&      some\\
     rain&      75&     80&     false&  some\\\hline
     sunny&    69&     70&     false&  lots \\
     sunny&    75&     70&     true&   lots\\
     overcast&  83&     88&     false&  lots\\
     overcast&  64&     65&     true&   lots\\
     overcast&  72&     90&     true&   lots\\
    overcast&  81&     75&     false&  lots\\\hline
end{tabular}\end{center}}}
\footnotesize
begin{minipage}{.45\linewidth}
begin{verbatim}
ELECT class
ROM original
HERE outlook=
     'overcast'
ots
ots
ots
ots
end{verbatim}
end{minipage}\hfill
begin{minipage}{.45\linewidth}\begin{verbatim}
ELECT class
ROM original
HERE humidity >= 90
one
one
one
ome
ots
end{verbatim}
end{minipage}
 \caption{Attributes that select for golf playing behavior.}
label{fig:golfdata}
end{figure}
AR2 searches for a strong {\em select statement} that most {\em
hanges} the ratio of classes. To understand the concept of a {\em
trong select statement}, consider the log of golf playing behavior
een in \fig{golfdata}. In that log, we only play {\em lots} of golf
n $\frac{6}{5+3+6}= 43\%$ of the cases. To improve our game, we
ight search for conditions that increases our golfing frequency. Two
uch searches are shown in the bottom of \fig{golfdata}. In the case
f {\tt outlook=overcast}, we play {\em lots} of golf all the time.
n the case of {\tt humidity $\ge$ 90}, we only play {\em lots} of
olf in 20\% of the cases. The net effect of these two select
tatements is shown in \fig{golft}.
begin{figure}
begin{center}
\small
begin{tabular}{p{0.7in}p{1.1in}p{1in}}
baseline$  & $outlook =overcast$
$humidity \ge 90$\\\\
begin{barenv}
setstyle{\sffamily} \setwidth{8}
    \setstretch{5}
 \setyaxis{0}{6}{2}
sethspace{1} \setnumberpos{down} \bar{5}{1}
\bar{3}{4}
\bar{6}{8}
end{barenv}&
begin{barenv}
setstyle{\sffamily} \setwidth{8}
\setstretch{5}
 \setyaxis{0}{6}{2}
sethspace{1} \setnumberpos{down} \bar{0}{1} \bar{0}{4} \bar{4}{8}
end{barenv}&
begin{barenv}
setstyle{\sffamily} \setwidth{8}
\setstretch{5}
 \setyaxis{0}{6}{2}
sethspace{1} \setnumberpos{down} \bar{3}{1} \bar{1}{4} \bar{1}{8}
end{barenv}\\\\~\\
multicolumn{3}{c}{LEGEND: \protect\legend1{none}
protect\legend4{some} \protect\legend8{lots}}
end{tabular}}\end{center}
\caption{Changes to golf playing behavior from the baseline.  }
label{fig:golft}
end{figure}
he {\tt WHERE} statements within a select statement can contain
onjunctions of arbitrary size. Exploring all such conjunctions
anually is a tedious task. TAR2 automatically finds the
trongest select statements; i.e., the statement that \emph{most}
elects for preferred behavior while \emph{most} discouraging
ndesirable behavior. TAR2 calls this strongest select statement the
`treatment'' since it is a recommended action for improving the
urrent situation. TAR2's
onfiguration file lets an analyst search for the best select
tatement using conjunctions of size 1,2,3,4, etc. Since TAR2's
earch is elaborate, an analyst can automatically find the {\em best}
nd {\em worst} possible situation within a data set. For example,
he select statements seen in \fig{golft} were learnt by TAR2 and
how the {\em best} and \emph{worst} possible situation for playing
\em lots} of golf.
AR2 processed \fig{tar2eg}.A by dividing the known outputs
nto ``preferred'' and ``undesired''
egions (here, ``preferred'' means lower costs and higher benefits
nd ``undesired'' means not preferred). With knowledge of that
ivision, TAR2 learnt a set of constraints that select for the
referred outcomes while avoiding the undesired regions. The goal of
AR2 is to {\em improve the mean} and {\em reduce the variance} in
he behavior of a system.
fig{tar2eg}.A shows 50,000 runs with DDP using mitigations
ompatible with  the constraints learnt by TAR2. Comparing
fig{tar2eg}.A with\fig{tar2eg}.B, we see that the variance in
ehavior has indeed been greatly {\em reduced} while {\em decreasing}
ean costs and {\em increasing} mean benefits.
AR2 generated \fig{tar2eg}.B using only a small subset of the
vailable risks mitigations. TAR2 made recommendations on only
\frac{1}{3}rd$ of the 99 mitigations available in this DDP models.
urther details on this use of TAR2 on a DDP dataset are found in
eference~\cite{fea02a}.
subsection{Drawbacks with TAR2}
fig{tar2eg}.B shows that it is possible to use DDP models to
ptimize risk mitigation actions for complex systems, using only a
\em small subset} of the available options. However, in two aspects,
he TAR2 experiment was a failure: \bi \item {\em The runtime
roblems:} TAR2 is too slow. The DDP model had to be executed 50,000
imes to learn the constraints that generated Figure 2b. This runtime
s too long to support interactive argument support. Worse still,
igger DDP models would take even longer to execute. Clearly, a
aster method is required. \item {\em The hiding problem}. TAR2's
utput can hide important details. Recall from \fig{tar2eg}.B that
here exists a cluster of results that are the best TAR2 can find.
hile any point in those clusters are the best TAR2 can offer,
djacent points in the cluster may represent very different
itigations, some of which are more acceptable to the users than
thers.\ei
AR2 ran slow since it sampled a large run where mitigations were
elected randomly.  Perhaps some other search might be more
ppropriate? In the sequel, we will discuss the merits of TAR2's
earch vs {\em simulated annealing}.
s to the {\em hiding problem}, we believe it is best addressed as a
\em quantitative value} problem. A  limitation of the DDP ontology
s that it asks a set of experts to agree upon some numeric quantity
using a number between 0.0 and 1.0) to rate various relationships:
.g., the impact of risks on objects, the effect of mitigations on
isk, etc. In our experience, these experts have been able to do
his.  However, it is clear that the resulting model is less robust
han the numeric values may suggest.
t is our hypothesis that such experts may be more comfortable
greeing upon a probability distribution that represents the impact
f a risk on an objective or the effect of a mitigation on a risk.
hat is, they would be given a small number of possible distributions
\em having previously been informed about the characteristics of
ach} and asked to pick among them.  They would also have to pick the
ppropriate parameters for each -- e.g. mean and standard deviation
or a normal distribution.
he thought is that, although this is more information for the
xperts to agree upon, it might get agreement faster since they would
e recognizing that there is some embedded uncertainty in these
alues. agreeing upon a probability distribution that represents the
mpact of a risk on an objective or the effect of a mitigation on a
isk. That is, they would be given a small number of possible
istributions (having previously been informed about the
haracteristics of each) and asked to pick among them.  They would
lso have to pick the appropriate parameters for each -- e.g. mean
nd standard deviation for a normal distribution.  Our thinking is
hat, although this is more information for the experts to agree
pon, it might get agreement faster since they would be recognizing
hat there is some embedded uncertainty in these values.
subsection{Is Simulated Annealing Better Than TAR2?}
begin{figure}
begin{center}
includegraphics{costs.eps}\\
includegraphics{benefits.eps}
end{center}
caption{Comparison of TAR2 and simulated
nnealing.}\label{fig:comparsa}
end{figure}
fig{comparsa} compares TAR2 and simulated annealing. At each round X
shown on the x-axis), simulated annealing or TAR2 was used to
xtract key decisions from a log of runs of a DDP model. A new log is
enerated, with the inputs constrained to the key decisions found
etween round zero and round X. Further rounds of learning continue
ntil the observed changes on costs and benefits stabilizes.
t is insightful to compare the results from TAR2 and simulated
nnealing: \bi \item  As seen in \fig{comparsa}, simulated annealing
nd TAR2 terminate in (nearly) the same cost-benefit zone.
item
Simulated annealing did so using only 40\% of the data needed by TAR2; i.e. while TAR2 needed 50,000 runs of DDP, the simulated annealing method needed only 20,000.
item  The bad news is that, while TAR2 proposed constraints on 33\%
f the mitigations, simulated annealing proposed actions on 100\% of
he mitigations. Such a result is consistent with the nature of
imulated annealing- this search is a global search through all
ptions. Hence, it tends to propose solutions to a large part of the
odel. \ei
n summary, the directed search of simulated annealing needs less
ata than TAR2, but in doing so, we lose the main advantage of TAR2;
.e. no drastic reduction in the space of options.
section{Conclusions and Future Work}
ur current work has produced a set of tools to support decisions
hat need to be made by software engineers early in the software
evelopment life cycle.  The DDP tool supports a model of requirements,
isks (things that may cause requirements not to be attained), and
itigations (activities to reduce these risks.)  This tool has been supplemented
ith an automated search mechanism (simulated annealing) that automates
he process of finding near optimal sets of decisions in the large space
f possible mitigating actions.  We have also introduced some clustering
bilities into DDP.  This allows a visual representation of sets of decisions
hat are that are self-similar within each cluster, and significantly
istinct between cluster.  In a parallel development, we have been exploring
he use of machine learning to identify those critical decisions that
ave an especially strong impact on cost and effectiveness.
subsection{Future Work: STAR1= simulated annealing + TAR2}
n the future, we plan to explore additional ways of modeling
equirements and risk early in the lifecycle, to more completely
ntegrate this existing set of tools, and to further test this integrated
ackage on actual projects at JPL or other NASA laboratories.
or example, the directed search of SA needs less data than TAR2, but
n doing so, we lose the main advantage of TAR2; i.e. no drastic
eduction in the space of options.  Perhaps we can get the best of
oth approaches.
ur research is exploring combining the advantages of TAR2 (the
election of a small number of critical decisions) with SA (faster,
irected search and an exploration of a larger space of
ossibilities). The ``jumps'' in simulated annealing are generated by
utating the best solution seen so far. In traditional SA, these
utations are selected at random. In our proposed approach, we would
un a contrast set learner in parallel with the SA to build up a
robability profile on settings that were most associated with worse
olutions. The mutation sub-routine of the SA would then be modified
o avoid mutations that include settings from the worst solutions.
ur analogy for this process is that of a rocket flying down towards
ome preferred solution. SA is the {\em gravity} that pulls the
ocket down faster while the contrast set learning is the {\em
ooster} than thrusts the rocket away from undesired situations.
pecifically, our goals are: \bi \item  Implement STAR1, a
ombination of SA (or other AI search algorithms) and TAR2, and
ntegrate the result with DDP \item Tune the STAR1 such that it such
hat it terminates in $<10$ seconds (i.e. in time to interact with
ome active debate on some part of a DDP model). \item   Augment this
ntegrated tool (STAR1) with a decision clustering tool \item Improve
he modeling of risk in DDP through probability distributions \item
est this supplemented version of DDP during live debates on system
ptions by JPL analysts. \ei
bibliographystyle{abbrv}
\footnotesize
bibliography{../../refs}}
end{document}