In \tion{stability} we looked at the variability between
each trial; in this section we will look at the variability
within the trials.
Recall from \tion{solutionForm} that a partial
description actually forms a set of solutions,
comprising all solutions that pass the treatments in
the partial description.
These solutions will naturally be scored
differently according to the objective function.
A key question then is:
what amount of variability can we expect from
a partial description returned by ITL?

To address this question we present some results that
show the estimated variance based on the standard deviation
of the points generated during each iteration.
The figures in this section show the average cost and benefit
side by side.
The averages were calculated in the same way as in
\tion{stability}, with a small amount of horizontal jitter
added to separate the ten different trials.
The standard deviation of the points in each iteration
are shown as error bars.
Also as in \tion{stability}, we will focus on the results
from the 500/75 combination and the \bore selector.
\tion{varianceApp} has a complete set of results.


\subsection{\aero}\label{sect:varAero}
\fig{varAero} displays
the per-iteration average of the 500/75 combination
for the \bore and \borep selectors.
The \wob selector (shown in \tion{varianceApp}), which 
has been shown to find lower quality solutions, also
has a much higher variance due to the partial description.
This is in line with the results from \tion{settings} and
\tion{stability} which also showed similar poor performance
(according to the criteria used in those sections)
of the \wob selector.


\begin{figure}[t]
\centering
\begin{tabular}{cc}
\multicolumn{2}{c}{\bore} \\
cost & benefit \\
\includegraphics[width=2.395in]{Figures/DDP_figs/costAERO-BOR-500-75.png} &
\includegraphics[width=2.395in]{Figures/DDP_figs/benAERO-BOR-500-75.png} \\
\multicolumn{2}{c}{\borep} \\
cost & benefit \\
\includegraphics[width=2.395in]{Figures/DDP_figs/costAERO-BORPRIME-500-75.png} &
\includegraphics[width=2.395in]{Figures/DDP_figs/benAERO-BORPRIME-500-75.png} \\
\end{tabular}
\caption[Cost and\ Benefit in the \aero model with $M/N$ values set to 500/75]
{Cost and\ Benefit in the \aero model with $M/N$ values set to 500/75.
Error bars represent the standard deviation of the points in each iteration.}
\label{fig:varAero}
\end{figure}

Looking at the \bore and \borep results we see some of the patterns
seen in \tion{settings} and \tion{stability} repeated and some contradicted.
Using the \bore selector it appears the least variable trials
are from the combinations where $N$ = 75.
Both the $M$ = 500 and $M$ = 300 cases
(the $M = 300$ cases can be found in \tion{varianceApp})
have only one outlier that takes 3-5 more iterations
to converge to the average benefit seen in the other nine trials.
But after these ``late'' trials do converge they have  
small standard deviations similar to the other trials.
This is similar to the pattern we saw in \tion{settings} and \tion{stability}
where $N$ = 25 combinations are underperformers compared to $N$ =75 combinations.

Something that we have not seen in previous sections is the apparently
superior performance of the \borep selector for $M$ = 500 combinations.
\fig{varAero} shows faster convergence in the average benefit,
as well as lower variance.
It is not clear why this is the case for this particular model,
because this pattern is not seen in the other models
(discussed below).



\subsection{\holo}\label{sect:varHolo}
For the \holo model all three selectors show a similar pattern of
decreasing variance, therefore we will only display the \bore
selector.
\fig{varHoloBore} shows the 500/75 and 500/25 combinations.
This figure shows an extremely low variance in the cost,
with a decreasing, but higher variance in the benefit,
for the 500/75 combination.
The 500/25 has a higher variance in both the average cost and 
average benefit.
The $M=300$ cases are similar and the $M=100$ cases show a significantly
higher variance, particularly in the average benefit.

\begin{figure}[t]
\centering
\begin{tabular}{cc}
\multicolumn{2}{c}{500-75} \\
cost & benefit \\
\includegraphics[width=2.395in]{Figures/DDP_figs/costHOLO-BORPRIME-500-75.png} &
\includegraphics[width=2.395in]{Figures/DDP_figs/benHOLO-BORPRIME-500-75.png} \\

\multicolumn{2}{c}{500-25} \\
cost & benefit \\
\includegraphics[width=2.395in]{Figures/DDP_figs/costHOLO-BORPRIME-500-25.png} &
\includegraphics[width=2.395in]{Figures/DDP_figs/benHOLO-BORPRIME-500-25.png} \\
\end{tabular}
\caption[Cost and\ Benefit in the \holo model with the \bore selector]
{Cost and\ Benefit in the \holo model with the \bore selector.
Error bars represent the standard deviation of the points in each iteration.}
\label{fig:varHoloBore}
\end{figure}


\subsection{\cob}\label{sect:varCob}
The cob model seems to have a particularly low variance inherent in
its structure.
This is similar to the high stability we saw in the \cob
model in \tion{stability}.
For all selectors and all combinations the variance in both 
the average cost and benefit quickly diminishes and by
the fourth to sixth iteration is quite low.
The \bore and \wob selector are shown in \fig{varCob} to demonstrate 
how the usually underperforming \wob selector has the same
level of variance as the \bore selector.
Both selectors show a steadily reducing variance as
the search proceeds, with the \cob selector taking just 1-3
more iterations for the variance in the average cost to reach
the level seen in the \bore selector.


\begin{figure}[t]
\centering
\begin{tabular}{cc}
\multicolumn{2}{c}{\bore} \\
cost & benefit \\
\includegraphics[width=2.395in]{Figures/DDP_figs/costCOB-BOR-500-75.png} &
\includegraphics[width=2.395in]{Figures/DDP_figs/benCOB-BOR-500-75.png} \\

\multicolumn{2}{c}{\wob} \\
cost & benefit \\
\includegraphics[width=2.395in]{Figures/DDP_figs/costCOB-WOB-500-75.png} &
\includegraphics[width=2.395in]{Figures/DDP_figs/benCOB-WOB-500-75.png} \\
\end{tabular}
\caption[Cost and\ Benefit in the \cob model with $M/N$ values set to 500/75]
{Cost and\ Benefit in the \cob model with $M/N$ values set to 500/75.
Error bars represent the standard deviation of the points in each iteration.}
\label{fig:varCob}
\end{figure}

The familiar patterns, with regard to the $M/N$ values, 
from \tion{settings} and \tion{stability} are 
seen in the \cob results, although with reduced significance.
That is, we see higher variance in lower values of $M$ and in
the lower values of $N$, but the 300/75 combination performs slightly better
than the 500/25 combination.


%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "../thesis"
%%% End: 
