\subsection{Performance Score}\label{tion:perform}
As POM2 requirements are completed, they are 
%At the end of an iteration, all completed requirements are 
moved to a ``done'' set.
%and requirements left in the plan are passed to the next iteration.
%As each new requirement is added to the ``done'' set, statistics are kept on
%the team's performance:
%\begin{enumerate}
%\item Sum of costs (of completed requirements)
%\item Sum of values (of completed requirements) 
%\end{enumerate}
\begin{figure}\small
\begin{center}
Very low dynamism: $\sigma=0, \lambda=0$

\includegraphics[width=2in]{charts/fr-v-sc-vl.pdf}

Medium dynamism: $\sigma=0.15, \lambda=0.015$

\includegraphics[width=2in]{charts/fr-v-sc-mi.pdf}

High  dynamism: $\sigma=2, \lambda=0.2$

\includegraphics[width=2in]{charts/fr-v-sc-vh.pdf}
\end{center}
\caption{Distribution of Performance scores, controlling only $\sigma$ and $\lambda$ inputs,
%while choosing at random for other model inputs.
}\label{fig:dist} 
\end{figure}
The $Sum$ $of$ $Costs$ and $Sum$ $of$ $Values$ are plotted on a (X,Y) plane.
%we can visualize the time-varying performance of the team:
When the project is finished, the plot can be compared to an
\textit{optimal frontier}, obtained by sorting all the ``done'' requirements
using the final \(\frac{value}{cost}\) for all those requirements.  
Note that
%Note that the sort used
%to generate the optimal frontier relies on information not available 
%halfway through the simulation.
% (specifically, the 
%\(\frac{final\; cost}{value\; of\; each\;  requirement}\)).  
%The Optimal Frontier works on the same set of requirements, except that those requirements \(\frac{cost}{value}\) pairs are 
%updated to their new value before the simulation starts. 
no method can do better than the optimal frontier.

Scoring works as follows.  If a project finishes after completing, say
100 requirements, then we would score the run as follows:
\begin{enumerate}
\item[1] Take the final \(\frac{value}{cost}\) curve for the project.
\item[2] Divide that by the optimal frontier \(\frac{value}{cost}\) curve.
\end{enumerate}

% & \(\sigma = 0.00\) & \(\sigma = 0.15\) & \(\sigma = 2.00\) \\

%\fig{dist} is a distribution of 10,000 runs of POM2 using the Performance Score described above.
%The performance scores generated in this way are shown in \fig{dist}. 
%Note that, in those results, the agile projects that sort using just $value$ always perform
%worse than the agile2 projects that use $value/cost$ (the blue plots always score lower than the red plots). This is an interesting
%result since, according to Cao and Ramesh~\cite{lCao_bRamesh}, sorting on just $value$ is a common agile practice.
%In traditional Agile development, sorting is on just $value$\cite{lCao_bRamesh}
%In a personal communication, Alistair Cockburn has stressed the need adjust priorities using value $and$ cost.
%This is refered to as Agile2.

%Two interesting features of \fig{dist}, are:
%\bi
%\item 
%Agile methods do not perform better than plan-based when dynamism is low (see top plot).
%\item
%The median performance of plan-based {\em decreases} as dynamism {\em increases}.
%\ei
%That is, while there is no advantage of use agile 
%for low dynamism situations, 
%agile methods can adapt to increasing dynamism (while plan-based cannot).
%This leads to the following conclusion:
%organizations should adopt agile2 processes as their default methodology.

\section{Searching}
\fig{dist} is a distribution of 10,000 runs of POM2 scored as described above. Note that Agile methods
maintain equal or better performance when compared to Plan-based methods.
The \fig{dist} results are interesting, but they are a two-dimensional summary of a 
ten-dimensional space.
%\bi
%\item one output value; 
%\item one choice of prioritization policy; 
%\item eight input values. 
%\ei
%It is possible that that the general finding that agile2 works as well, or better
%than plan-based does not hold for regions within that 10-D space.
Before adopting the above conclusion,
we need to find any special regions in the 10-D space of POM2.

We define these regions as follows: Given a model with $inputs$ to a model that are  a set of
$feature_i=range_i$, find the smallest subsets that most change  the model $output$.
Each such subset is special region since it is here
where  the score changes most.
Our preferred tool for this analysis is 
KEYS~\cite{mens08}:
a greedy search and a Bayesian ranking
method called BORE. 
%The greedy
%search starts to explore the domain. 
Assume that the model has $N$
accessible states which are the inputs to the model (in our case, our sates are assignments to the Boehm and Turner scales). 
These inputs
are randomized from a distribution $\{X_i, i=1 \dots N\}$. A
collection of inputs to the model is called a treatment. 
\begin{equation}
\{x_1, x_2,\dots , x_N\} = treatment
\end{equation}
After $M$ number of initial samples, a second phase starts and one
of the inputs is fixed to a desired range. This is performed
for each of the input variables (the Boehm scales)
until the stopping criteria is met. KEYS stops if the improvement
from the previous round is less than 5\%. The range is selected
using the BORE heuristic. At the next run the $treatment = \{fixed\}
\bigcup \{randomized\}$. 
%KEYS' pseudo-code is shown in \fig{keys}.

%\begin{figure}[!h]\small\begin{algorithmic}
%\WHILE{$Fixed\_Inputs \neq Total\_Inputs$}
%\FOR{I = 1 to N}
%\STATE{$Selected[1...(I-1)] := \mbox{best decisions so far}$}
%\STATE{$Guessed := \mbox{random settings to the remaining mitigations}$}
%\STATE{$Input := Selected \cup Guessed$}
%\STATE{$Scores	:= Score(Input)$}
%\ENDFOR
%\STATE{$Top\_Range := Bore(Scores)$}
%\STATE{$Fixed\_Inputs \mbox{~+=~} 1$}
%\STATE{$Selected(Fixed\_Inputs) =Top\_Range$}
%\ENDWHILE
%\RETURN $Selected$
%\end{algorithmic} \caption{KEYS pseudo-code}\label{fig:keys}
%\end{figure}
%\addtocounter{footnote}{1}
\begin{figure*}
%\renewcommand{\baselinestretch}{0.6} 
\begin{center}
{\scriptsize \input{tables}}
\end{center}
\caption{The top table shows the division of each
numeric inputs divided into ten bins. 
The rest of this figure   shows
the percent frequencies with which a 
range appears in the treatments found by 1000 runs of KEYS. 
%Inputs are only shown when chosen in a significant number of cases $^2$.
}
%Results taken from 1000 repeats for each pair
%of policy/dynamism.
\label{fig:1000}
\end{figure*}
%At its core, KEYS uses the BORE heuristic to rank ranges.  BORE
%takes $N$ inputs that have been scores with outputs.  For this
%study, we used the performance scores generated by the method of
%\tion{perform}.  It assumes that any numeric values have been
%converted into a set of bins.  For our work, we divide the Boehm
%Turner scales into ten equal width ranges.

%These $N$ 
%inputs are then split into two groups - the 10\% best (with highest value) and the 90\%
%rest \cite{clarkr05}. The probability that a input range is found in
%$best$ is then determined by using Bayes theorem. The theorem uses
%evidence E and a prior probability P(H) for hypothesis $H \in \{best,
%rest\}$, to calculate a posteriori probability $P(H|E) = P(E|H)P(H)
%/ P(E)$. When applying the theorem, likelihoods are computed from
%observed frequencies (hereafter, $freq$). These likelihoods (hereafter, $like$ ) are
%then normalized to create probabilities. This normalization cancels
%out $P(E)$ in Bayes theorem. For example, after N = 10, 000 runs
%are divided into 1,000 best solutions and 9,000 rest, the value for
%$X_i$ is in a certain range might appear 10 times in the best
%solutions, but only 5 times in the rest. Therefore:

%\[\small
%\begin{array}{r@{~}l@{~}l}
%P(best)& =& 1000/10000 = 0.1  \\
%P(rest)& =& 9000/10000 = 0.9  \\
%freq(E|best)&=&10/1000 = 0.01  \\
%freq(E|rest)&=&5/9000 = 0.00056  \\
%like(best|E)&=&freq(E|best) · P(best) = 0.001  \\
%like(rest|E)&=&freq(E|rest) · P(rest) = 0.000504 \\
%P(best|E)&=&\frac{like(best|E)}{like(best|E) + like(rest|E)} = 0.66
%\end{array}
%\]
%With this setup the model is sensitive to small frequencies. For example, the computed value of 0.66 hides the
%fact that the range was only seen 15 times in 10,000 samples.
%To penalize such samples, we introduce a {\em support} term into the calculations. This support term should be
%large if a range is frequent; i.e. the $like$ value suffices as a support measure. Hence, when KEYS looks for
%ranges that most effect output, it uses:
%\begin{equation}\footnotesize
%P(best|E)*support(best|E) =\frac{like(best|E)^2}{like(best|E) + like(rest|E)}
%\end{equation}
%
%In other work (under review, submitted to ASE 2009) we
%show that BORE+KEYS  is surprisingly powerful at 
%finding these regions.
%Empirically, we have shown that
%KEYS can out-perform an interesting range of alternate
%search engines such as simulated annealing, A-STAR, and MaxWalkSat.
%
%\begin{figure}
%\caption{Frequency vs score original - Very High}
%\label{figure:fr-v-sc-vh}
%\end{figure}
%Frequency vs score original - Very High see Figure \ref{figure:fr-v-sc-vh}
%
%%\begin{figure*}[t]
%%\input{tables/ranges}
%%\caption{Attribute Ranges}
%%\label{table:att-ranges}
%%\end{figure*}
%%Attribute Ranges by Bin see \ref{table:att-ranges}
%%
%%\begin{figure*}[t]
%%\input{tables/verylow/PB}
%%\caption{Very Low Dynamism - Plan Based}
%%\label{table:vl-pb}
%%\end{figure*}
%%Very Low Dynamism for Plan Based see Figure \ref{table:vl-pb}
%%
%%\begin{figure*}[t]
%%\input{tables/middle/PB}
%%\caption{Middle Dynamism - Plan Based}
%%\label{table:mi-pb}
%%\end{figure*}
%%Middle Dynamism for Plan Based see Figure \ref{table:mi-pb}
%%
%%\begin{figure*}[t]
%%\input{tables/veryhigh/PB}
%%\caption{Very High Dynamism - Plan Based}
%%\label{table:vh-pb}
%\end{figure*}
%Very High Dynamism for Plan Based see Figure \ref{table:vh-pb}
%
%
%
%\begin{figure*}[t]
%\input{tables/verylow/AG2}
%\caption{Very Low Dynamism - Agile 2}
%\label{table:vl-ag2}
%\end{figure*}
%Very Low Dynamism for Agile 2 see Figure \ref{table:vl-ag2}
%
%\begin{figure*}[t]
%\input{tables/middle/AG2}
%\caption{Middle Dynamism - Agile 2}
%\label{table:mi-ag2}
%\end{figure*}
%Middle Dynamism for Agile 2 see Figure \ref{table:mi-ag2}
%
%\begin{figure*}[t]
%\input{tables/veryhigh/AG2}
%\caption{Very High Dynamism - Agile 2}
%\label{table:vh-ag2}
%\end{figure*}
%Very High Dynamism for Agile 2 see Figure \ref{table:vh-ag2}
%%
%
%
%\begin{figure}
%\input{charts/k-vs-score-mww-by-policy-vl}
%\caption{Win Loss Tie chart split by policy and indexed by iteration for Very Low Dynamism}
%\label{fig:mww-p-vl}
%\end{figure}
%Win Loss Tie chart broken up by policy and indexed by iteration count for Very Low Dynamism \ref{fig:mww-p-vl}
%
%\begin{figure}
%\input{charts/k-vs-score-quartile-vl}
%\caption{Quartile chart for Very Low Dynamism}
%\label{fig:quartile-vl}
%\end{figure}
%Win Loss Tie chart broken up by iteration count for Very Low Dynamism \ref{fig:quartile-vl}
%
%
%
%\begin{figure}
%\input{charts/k-vs-score-mww-by-policy-mi}
%\caption{Win Loss Tie chart split by policy and indexed by iteration for Medium Dynamisim}
%\label{fig:mww-p-mi}
%\end{figure}
%Win Loss Tie chart broken up by policy and indexed by iteration count for Medium Dynamisim \ref{fig:mww-p-mi}
%
%\begin{figure}
%\input{charts/k-vs-score-quartile-mi}
%\caption{Quartile chart for Medium Dynamisim}
%\label{fig:quartile-mi}
%\end{figure}
%Win Loss Tie chart broken up by iteration count for Medium Dynamisim \ref{fig:quartile-mi}
%
%
%
%\begin{figure}
%\input{charts/k-vs-score-mww-by-policy-vh}
%\caption{Win Loss Tie chart split by policy and indexed by iteration for Very High Dynamisim}
%\label{fig:mww-p-vh}
%\end{figure}
%Win Loss Tie chart broken up by policy and indexed by iteration count for Very High Dynamisim \ref{fig:mww-p-vh}
%
%\begin{figure}
%\input{charts/k-vs-score-quartile-vh}
%\caption{Quartile chart for Very High Dynamisim}
%\label{fig:quartile-vh}
%\end{figure}
%Win Loss Tie chart broken up by iteration count for Very High Dynamisim \ref{fig:quartile-vh}
%
