\chapter{Requirements Engineering}\label{chap:casestudies}
The main case study presented in this thesis uses ITL
to optimize cost/benefit models.  In this chapter
we describe our models and their execution framework, DDP,
(\tion{ddpDes}),
investigate the effect of different parameter
settings on the performance of extreme sampling
(\tion{settings}),
investigate the stability of the partial descriptions
(\tion{stability}),
investigate the variance of the partial descriptions
(\tion{variance}),
compare the performance of the original discretization
policy with extreme sampling (\tion{boreAndiD}),
compare the performance of extreme sampling
with the original simulated annealer in DDP (\tion{boreAndSA}),
and summarize the lessons learned from this study (\tion{ddpConclusions}).

\section{Defect Detection and Prevention}\label{sect:ddpDes}
\input{casestudies/ddpdescription}

\section{Investigating Different Extreme Sampling Policies}\label{sect:settings}
\input{casestudies/settings}

\section{Stability of Repeated Trials}\label{sect:stability}
\input{casestudies/stability}

\section{Variance of Partial Descriptions}\label{sect:variance}
\input{casestudies/variance}


\section{Extreme Sampling Performance Compared to Diagonal Striping}\label{sect:boreAndiD}
\input{casestudies/boreid}


\section{Extreme Sampling Performance Compared to Simulated Annealing}\label{sect:boreAndSA}
\input{casestudies/boreandSA}

\section{Conclusions from the DDP Studies}\label{sect:ddpConclusions}
This chapter has used requirements engineering to investigate
several important characteristics of ITL.
First we investigated different possible policy settings to
extreme sampling, a key part of ITL.
Then we compared ITL's performance with extreme sampling
to its performance with diagonal striping.
Lastly, we compared the performance of ITL with extreme sampling
to simulated annealing, a well studied and frequently used 
metaheuristic search technique.

There are several conclusions we can draw from all
these experiments.
Extreme sampling works best with the \bore selector.
While we hoped that \borep or \wob could be used to
reduce the amount time needed by the learner,
by reducing the number of instances trained on,
both \borep and \wob caused a degradation in the
performance of ITL when compared to the \bore 
selector.
Our intuition that the \wob selector might perform
better by highlighting the difference between the instances with
a high objective score and those with a low objective score
also turned out to be incorrect.

Learning on large batch sizes positively effects
solution quality when the search is iteration limited,
but if the search is limited by the number of times the
objective function can be evaluated, smaller batches
positively effects solution quality.
The experiments in this chapter did not use a wide enough range of
$M$ values to hint at the value of $M$ at which ITL's performance
stops increasing with increasing values of $M$.

We found that reducing the number of instances labeled
good negatively affects solution quality.
Extreme sampling needs a certain percentage of instances in 
each iteration to be labeled good.
The ratio of $N$ to $M$ appears to be at least 15\% to 25\%
(75/500 to 75/300).
The experiments in this chapter did not use a wide enough range of
$N$ values to hint at the upper limit to this ratio.

Sections \ref{sect:stability} and \ref{sect:variance} demonstrated
that ITL with extreme sampling has both stable performance
(restarting a search does not significantly change the
performance of the method) 
and that the variance inherent in partial descriptions 
is low after several iterations.

In \tion{boreAndiD} we demonstrated the clear superiority
of the \bore version of extreme sampling
to our previous discretization method, diagonal striping.
Averaged over the two models, \bore found a solution 
with an objective score 161\% of the average found by diagonal striping.

Finally \tion{boreAndSA} showed that extreme sampling can find
higher, equal, or lower quality solutions than simulated
annealing, but in many fewer objective function evaluations.
Averaged over the three models, ITL found a solution 
with an objective score 98.6\% of the average found by the simulated
annealer.

The next set of case studies have to do with the SPY framework.
Like ITL, SPY finds range restrictions to model input variables
by using a treatment learner.
However, SPY takes a different approach to integrating the search
algorithm with the model.
The models used by SPY are written in the SPY language, 
which was specifically designed to make the model-learner
interface as smooth as possible.
These studies highlight a problem we discussed earlier in 
\tion{rethinking}; using a nearly continuous objective function
is critical for a metaheuristic search to be successful.
\Chap{properties} will use SPY to validate temporal 
properties in NASA flight models and
\chap{others} will investigate restricting the behavior modes of
biomathematical models.

%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "../thesis"
%%% End: 
