The last section showed the performance of our 
different trials aggregated across all ten trials.
But are we sure that ITL produces stable results?
If, in a time sensitive setting, we run only one trial, can we
be sure that the solution found by this trial won't
be significantly outperformed by another run.
In this section we present the results previously discussed
by displaying the {\it search trajectories} for all ten trials.
A search trajectory is the path the search takes through
the output space.
The outputs from DDP are cost and benefit, so the
search trajectory can be plotted as 2-D line.
The x-axis is the normalized mean cost and 
the y-axis is the normalized mean benefit.
These means were calculated by averaging the
costs and benefits of the instances that were generated
by DDP each iteration.  
Hence each point represents one iteration.
The lines are the successive iterations from a single trial.

Since we can not aggregate this data across models or selectors,
as we did in the last section, this section will not
present the results of all parameter-model combinations.
Instead we will focus on the trials with $M/N$ values of
500/75 and the \bore selector, since these were the best
combinations found in \tion{settings}.
When there is something interesting to comment on,
other combinations or selectors will be discussed.
See \tion{stabilityApp} for a complete listing
of the results from this section.

The rest of this section will break down the results
by model, first \aerons, then \holo and finally \cobns.

%At this time we have not developed a statistical measure of 
%how ``close'' two trials are.
%Rather it is left to the reader to subjectively judge
%how close the trajectories and end points are to each other.

\subsection{\aero}
First we present some results from the \aero model.
We display the 500/75 and 500/25
trials using the \bore selector in \fig{aeroBore}.
First note that both combinations shown find the same
approximate benefit ceiling, slightly above .55 units.
But the superior performance of 500/75 is evident.
More trials find a lower cost, between .42 and .44 units,
and if you look closely at the point (.45,.20) on
the 500/25 graph there is an outlier which did not
find the benefit ceiling (the trial represented by the plus sign).

\begin{figure}
\begin{tabular}{cc}
500-75 & 500-25 \\
\includegraphics[width=2.75in]{Figures/DDP_figs/argandAERO-BOR-500-75.png} &
\includegraphics[width=2.75in]{Figures/DDP_figs/argandAERO-BOR-500-25.png} \\
\end{tabular}
\caption{Stability of cost and benefit in the \aero model with the \bore selector}
\label{fig:aeroBore}
\end{figure}


Examining the results from all combinations with the \bore selector,
we see the same pattern as the performance based comparison
demonstrated in \tion{settings}.
Larger values of $M$ show less variability run-to-run than do
smaller values.
But a large value of $M$ coupled with a small value of $N$
(like the 500/25 graph in \fig{aeroBore})
does poorly when compared to medium values of $M$ coupled with large
values of $N$
(specifically, the 300/75 trials show less variability than
the 500/25 trials, see \fig{aeroBoreApp}).
We can also see a model specific pattern in these comparisons.
The final average benefit of the partial descriptions is very similar
for almost all $M/N$ combinations;
the performance difference in the trials comes almost entirely from
their differing costs.


\fig{aeroWob} displays the same $M/N$ settings as \fig{aeroBore}, 
but from trials with the \wob selector.
\fig{aeroWob} clearly shows the inferior performance 
of the \wob selector that we discussed in \tion{settings}.
These graphs also clearly shows that in addition to inferior performance,
the \wob selector also has a lower stability in solution quality.
The end of points of the trajectories have significant outliers
for all parameter settings shown, even for the parameter combination
that showed the most stability with the \bore selector (500/75).
The difference between 500/75 and 500/25 is also greater with the
\wob selector, suggesting that \wob is more sensitive to changes
in the $M/N$ values.


\begin{figure}
\begin{tabular}{cc}
500-75 & 500-25 \\
\includegraphics[width=2.75in]{Figures/DDP_figs/argandAERO-WOB-500-75.png} &
\includegraphics[width=2.75in]{Figures/DDP_figs/argandAERO-WOB-500-25.png} \\
\end{tabular}
\caption{Stability of cost and benefit in the \aero model with the \wob selector}
\label{fig:aeroWob}
\end{figure}


\subsection{\holo}
There was much less variation between the three different selectors
in the \holo model.
Therefore, we only present the results of using the \bore
selector.
\fig{holoBor} shows the trials that used $M/N$ combinations
of 500/75 and 500/25 with the \bore selector.
The endpoints of the 500/75 trials are much closer together
and this is a pattern repeated for all values of $M$ in the \holo
model.
Notice that trajectories follow different paths through the cost/benefit
plane, unlike the trajectories in \fig{aeroBore} which all followed the
same path.
All trajectories lowered the average cost during each iteration,
but some initially raised the average benefit, while some lowered
the average benefit.

\begin{figure}
\begin{tabular}{cc}
500-75 & 500-25 \\
\includegraphics[width=2.75in]{Figures/DDP_figs/argandHOLO-BORPRIME-500-75.png} &
\includegraphics[width=2.75in]{Figures/DDP_figs/argandHOLO-BORPRIME-500-25.png} \\
\end{tabular}
\caption{Stability of cost and benefit in the \holo model with the \bore selector}
\label{fig:holoBor}
\end{figure}



\subsection{\cob}
Finally we present some results from the \cob model.
The \cob model shows even less variability between the trials
than the \holo model, for different $M/N$ combinations
and for the different selectors.
To illustrate the similar variance, \fig{cobBore}
shows the $M/N$ values 500/75, 500/25, 100/50, and 100/25,
all from the \bore selector.
There is little to no difference between the 500/75 and 500/25
trials and while the 100/50 trials cluster at a slightly higher
cost, the trajectories do not show any more variability than
the trajectories for the 500/75 combination.
The 100/25 combination, which has been the worst performer in all comparisons
shown as far, shows a wider spread of trajectories,
but still clusters tightly after ten iterations.
In particular notice how tightly clustered all the trials are for 500/75.

\begin{figure}[ht]
\begin{tabular}{cc}
500-75 & 500-25 \\
\includegraphics[width=2.75in]{Figures/DDP_figs/argandCOB-BOR-500-75.png} &
\includegraphics[width=2.75in]{Figures/DDP_figs/argandCOB-BOR-500-25.png} \\
100-50 & 100-25 \\
\includegraphics[width=2.75in]{Figures/DDP_figs/argandCOB-BORPRIME-100-50.png} &
\includegraphics[width=2.75in]{Figures/DDP_figs/argandCOB-BORPRIME-100-25.png} \\
\end{tabular}
\caption{Stability of cost and benefit in the \cob model with the \bore selector}
\label{fig:cobBore}
\end{figure}



%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "../thesis"
%%% End: 
