So far in this chapter we have
investigated the performance characteristics of
extreme sampling using different parameters
and compared extreme sampling to our previous
discretization method, diagonal striping.
In this section we will compare ITL
to the previously used search strategy, simulated
annealing (see \tion{gradBased} for a description
of simulated annealing).

\paragraph{Original Search Technique Used by DDP}
As part of the DDP environment a simulated annealer
can be used to maximize the benefit/cost ratio.
In DDP's simulated annealer, the current best solution
is mutated as follows. 
The candidate solution is a boolean vector
representing whether to apply each mitigation.
Each boolean has a 10\% chance of flipping to the other boolean value.
The user specifies the number of objective function evaluations
desired so DDP can automatically set the cooling schedule.

\paragraph{Comparison of ITL to Simulated Annealing}
\fig{boreAndSA} shows the trajectories of DDP's simulated
annealer using 30,000 objective function
evaluations, and the trajectories of the best trial using
the \bore selector with $M/N$ values of 500/75.
(Hence 5000 objective function evaluations were done
during the \bore search.)
\ble{endPoints2} lists the normalized endpoints of
the six different searches in \fig{boreAndSA}.
(Remember that the normalized benefit/cost ratio
is our objective function.)
We can see that extreme sampling outperforms SA
in the \cob model, finds a solution with almost the same
value for the objective function for the \aero model,
and underperforms on the \holo model.

\begin{figure}
\centering
\includegraphics[width=3.05in]{Figures/aeroComparsion.png} \\
\includegraphics[width=3.05in]{Figures/holoComparsion.png} \\
\includegraphics[width=3.05in]{Figures/cobComparsion.png} \\
\caption{Comparison of \bore to SA}\label{fig:boreAndSA}
\end{figure}

Notice also that ITL seems to systematically find solutions
with higher benefit and cost (with the exception of the
cost of the solution found for the \cob model).
In personal communication with an experienced DDP user it
was suggested that this was due to the nature of a typical DDP
model.
A typical DDP model has some high-cost mitigations with large
benefits and many more low-cost mitigations that have small
benefits.
The simulated annealer can find a subset of low-cost mitigations
that together have a high benefit.  But this subset
will likely be much larger than a few treatments
(as previously discussed,
ITL uses treatments with a maximum size of five attribute-value pairs).
Since ITL only searches one treatment deep per iteration, it
cannot find this type of subset.
Instead the treatment learner finds the few mitigations that most greatly
effect the normalized ratio.
But obviously this search bias does not stop ITL from finding high quality solutions,
as discussed in the previous paragraph.




\begin{table} 
\centering
\begin{tabular}{rccccccc|}
\vline & \multicolumn{3}{c}{\bore}  & \vline & \multicolumn{3}{c}{SA} \vline \\
model \vline & cost & benefit & normalized ratio & \vline & cost & benefit & normalized ratio\\
\hline
\aero \vline & .366 & .569  & 1.55 & \vline & .302 & .482  & 1.60 \\
\holo \vline & .198 & .0869 & .439 & \vline & .153 & .0845 & .552 \\
\cob  \vline & .197 & .829  & 4.21 & \vline & .220 & .776  & 3.53 \\
\end{tabular}
\caption{Normalized benefit/cost ratios in \bore and SA}\label{table:endPoints2}
\end{table} 




%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "casestudies"
%%% End: 
