\chapter{Validating SPY on Biomathematical Models}\label{chap:others}
%\chapter{SPY: An Intergrated Model Development Environment}\label{chap:others}

The previous chapter introduced SPY and demonstrated
its ability to check certain types of properties
in NASA flight models.
We discussed how the nature of the model
inputs made validating a useful feature
of SPY difficult.
The nature of the range restrictions offered by SPY, 
\eg the pilot should never press the transfer switch,
were not practically useful.
To demonstrate SPY's ability to find useful range
restrictions we investigated a class of models
that have been developed and reviewed in research
fields outside of machine learning or metaheuristic
search.

SPY will attempt to find input range
restrictions to these new models, described
in \tion{biomath}, that confine
the behavior of the model outputs to specified
modes.


%The objective functions in the case studies using SPY
%are non-continuous because we use SPY to indentify
%behavior modes.
%Since a model either does or does not exhibit the
%desired behavior, the objective function is essentially
%boolean.
%This will cause problems for any search technique,
%but SPY can still be useful because of the nature of
%treatment learning.
%The attribute-value paris returned by the treatment learner in SPY 
%(tar4, which was described in \tion{tar3Details}) 
%are used to build range restrictions to the model input variables.
%So while we will see some unsuccessful searches in the next two
%chapters, when SPY does find the desired behaviors mode it
%still returns potentially useful range restrictions.


\section{Using ITL in Biomathematical Models}\label{sect:neuron}
\input{OtherCases/biomathmodels}


\section{Conclusions from Biomathematical Models}\label{sect:biomathConclusions}
Recall that in \tion{rethinking} we discussed
the desirability of using a continuous objective function.
The work presented in this section violates that advice.
In other words, if an instance was close to, but not in,
a portion of the parameter space that had a positive
worth value, the objective function would return 0 for that instance.
In the work presented in \chap{casestudies} we had
a continuous objective function, 
but this was our first experience trying to 
optimize what was essentially a boolean function.
We believe this is a general
problem when trying to identify types of behaviors
when there is no concept of being ``close to'' the
desired behavior.

This discontinuity of the objective function causes
what should be a heuristically driven
search to degrade to a purely random search.
If the objective function is discontinuous, then,
during the first few iterations, 
SPY won't be able to find treatments that restrict
future iterations to portions of the input space that 
have promising values.
In addition, the more discontinuous the objective,
\ie the smaller the region with a high objective evaluations,
the harder it will be for SPY (or any other stochastic search method)
to find these promising regions.
This can be seen in \fig{discont}.
All three lines are discontinuous, but the likelihood
of finding the optimal solution in the {\it impossible}
line is so much smaller than the likelihood of finding the
optimal solution in the {\it easy} line,
that no heuristic could outperform a purely random search.

\begin{figure}
\centering
\includegraphics[width=3.05in]{Figures/continuous.png}
\caption{Example of discontinuous functions}
\label{fig:discont}
\end{figure}


In some problem domains the objective function can be made
more continuous through domain specific knowledge.
In \tion{testing} we saw how some researchers trying to
automatically generate test cases modified the objective function
through symbolic analysis of the source code being tested.
This particular technique might be adapted to the experiments
presented in this chapter by including the value of
some the intermediate variables used in the objective function 
(\tion{worth})
as part of the value returned by the objective function.
For example, the original objective function for tonic bursting
was
\begin{equation*}
worth = \text{if} ~(fast > 10 ~\textbf{and} ~slow > 2) ~\text{then} ~1 ~\text{else} ~0
\end{equation*}
perhaps this could be modified to \\ 
\setstretch{1.3}
\begin{algorithmic}
\IF {$(fast > 10 ~\textbf{and} ~slow > 2)$} 
\STATE return 1
\ELSE
\STATE return -($|10 - fast|$ + $|2 - slow|$)
\ENDIF
\end{algorithmic}
\setstretch{2}
This function would return 1 when tonic bursting was present.
But when tonic bursting wasn't present it would return a negative number, 
whose value approaches zero as the intermediate values used by
the objective function, $fast$ and $slow$, approach the values
that indicate the presence of tonic bursting.
This type of modification might not work because we
can not be sure that small changes in the value of the independent
variables will only cause small effects on the intermediate
values used in the new proposed objective function,
\ie we can not be sure that the values of $fast$ and $slow$ are
approximately continuous in the independent variables.
Experimentation with objective functions of this type would reveal 
how useful this modification is.
Of course it is possible that this will be a domain
specific feature: some intermediate values may be
approximately continuous and some not depending on the model
being analyzed.


Regardless of the continuousness of the objective function,
the search bias of SPY's underlying learner, tar4,
can lead to over-restrictive bounds, of the type we saw in \tion{bioResults}.
To see why imagine that the true bound for some parameter $q$ is $(x,y)$.  
If the largest value for $q$ sampled
that exhibited the desired behavior was $r$ ($r<y$) and
the smallest value that didn't exhibit the desired behavior
was $s$ ($s>y$), no learner could reliably say anything
about the range $(r,s)$ (particularly if the objective function
is undifferentiable).
Because tar4  favors true-positives over true-negatives, 
it will always pick a range with an upper bound of $r$.
Then in future iterations the range $(r,y)$ will never
be sampled from, and the upper bound will have
no chance to increase to $y$.
This problem is similar to, but distinct from, getting trapped
in a local optimum.
While both may be solved by randomly sampling outside
the current range restrictions,
reporting overly restrictive bounds to the analyst
that still constrain the model to optimal behavior modes
is not the same as reporting bounds to the analyst
that constrain the behavior of the model to sub-optimal modes.
This problem in particular should be the focus of our future work.



%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "../thesis"
%%% End: 
