Executing models can pose comprehension problems. If used carelessly, analysts can become confused by an
information overdose. Hence, it is useful to augment a simulation engine with a search engine
that can find the most interesting regions within the model input/output space.

Augmenting simulation models with automatic AI search engines
has
been standard practice in our own research for several years~\cite{me00e,me02f,me01e,me03i,me07f,me08b,me08fh,me09a}, 
However, our 
reading of the software process modeling literature is
that is   usual practice for  analysts to perform semi-automatic or even
a manual sensitivity analysis of model input/outputs. 

Traditional sensitivity analysis assumes continuous, possibly even
linear equations.  For discrete sensitivity analysis, we prefer our
KEYS tools~\cite{mens08}.  KEYS is a combination of a greedy search and a Bayesian ranking
method called BORE (short for ``best'' or ``rest''). The greedy
search starts to explore the domain. Assume that the model has $N$
accessible states which are the inputs to the model (in our case, our sates are assignments to the Boehm and Turner scales). 
These inputs
are randomized from a distribution $\{X_i, i=1 \dots N\}$. A
collection of inputs to the model is called a treatment.
\begin{equation}
\{x_1, x_2,\dots , x_N\} = treatment
\end{equation}
After $M$ number of initial samples, a second phase starts and one
of the inputs is fixed to a desired range $X_i=M_j$. This is performed
for each of the input variables (in our case, the Boehm scales)
until the stopping criteria is met. Keys stops if the improvement
from the previous round is less than 5\%. The range is selected
using the a BORE heuristic (described below). At the next run the $treatment = \{fixed\}
\bigcup \{randomizede\}$.

\begin{figure}\small\begin{algorithmic}
\WHILE{$Fixed\_Inputs \neq Total\_Inputs$}
\FOR{I = 1 to N}
\STATE{$Selected[1...(I-1)] := \mbox{best decisions so far}$}
\STATE{$Guessed := \mbox{random settings to the remaining mitigations}$}
\STATE{$Input := Selected \cup Guessed$}
\STATE{$Scores	:= Score(Input)$}
\ENDFOR
\STATE{$Top\_Range := Bore(Scores)$}
\STATE{$Fixed\_Inputs \mbox{~+=~} 1$}
\STATE{$Selected(Fixed\_Inputs) =Top\_Range$}
\ENDWHILE
\RETURN $Selected$
\end{algorithmic} \caption{KEYS pseudo-code}\label{fig:keys}
\end{figure}
BORE takes $N$ inputs that have been scores with outputs. It assumes that any numeric values
have been converted into a set of bins.
For our work, we divide the Boehm Turner scales
into ten equal width ranges.

These $N$ 
inputs are then split into two groups - the 10\% best (with highest value) and the 90\%
rest \cite{clarkr05}. The probability that a input range is found in
$best$ is then determined by using Bayes theorem. The theorem uses
evidence E and a prior probability P(H) for hypothesis $H \in \{best,
rest\}$, to calculate a posteriori probability $P(H|E) = P(E|H)P(H)
/ P(E)$. When applying the theorem, likelihoods are computed from
observed frequencies (hereafter, $freq$). These likelihoods (hereafter, $like$ ) are
then normalized to create probabilities. This normalization cancels
out $P(E)$ in Bayes theorem. For example, after N = 10, 000 runs
are divided into 1,000 best solutions and 9,000 rest, the value for
$X_i$ is in a certain range might appear 10 times in the best
solutions, but only 5 times in the rest. Hence:

\[\small
\begin{array}{r@{~}l@{~}l}
P(best)& =& 1000/10000 = 0.1  \\
P(rest)& =& 9000/10000 = 0.9  \\
freq(E|best)&=&10/1000 = 0.01  \\
freq(E|rest)&=&5/9000 = 0.00056  \\
like(best|E)&=&freq(E|best) · P(best) = 0.001  \\
like(rest|E)&=&freq(E|rest) · P(rest) = 0.000504 \\
P(best|E)&=&\frac{like(best|E)}{like(best|E) + like(rest|E)} = 0.66
\end{array}
\]
With this setup the model is sensitive to small frequencies. For example, the computed value of 0.66 hides the
fact that the range was only seen 15 times in 10,000 samples.
To penalize such samples, we introduce a {\em support} term into the calculations. This support term should be
large if a range is frequent; i.e. the $like$ value suffices as a support measure. Hence, when KEYS looks for
ranges that most effect output, it uses:
\begin{equation}\footnotesize
P(best|E)*support(best|E) =\frac{like(best|E)^2}{like(best|E) + like(rest|E)}
\end{equation}
In other work (under review, submitted to ASE 2009), we
show that this very simple ranking heuristic can be surprisingly powerful.
KEYS can out-perform an interesting range of alternate
search engines such as simulated annealing, A-STAR, and MaxWalkSat.


