This chapter summarizes the thesis and suggest directions for future research.

\section{Summary of the Thesis}
Our goal in this thesis has been to evaluate the hypothesis that \emph{there are critical ranges of values for each attribute of a data set which can be used to select instances from the data set to be used as prototypes to improve the performance of nearest neighbor classifiers}. To this end we have presented a prototype learning scheme called CLIFF, that uses a ranking algorithm to create a criterion for each class in a data set. These criteria are then used to select instances as prototypes. CLIFF was then tested on eight(8) data sets and compared with three(3) PLS (CNN, MCS and PSC). We showed that CLIFF:

\be
\item has a time complexity of $O(n)$;
\item reduces training sets to a range of 9 to 15\%;
\item has $pd$ and $pf$ results which compares favorably with 1NN and other PLS in several standard data sets;
\item does not significantly increase the number of instances selected in the presence of noise as compared with other PLS;
\item reduces $brittleness$ substantially in most data sets used.
\ee

In response to the concern of the National Academy of Sciences which stated in a report \cite{09NAS} that, "With the exception of nuclear DNA analysis ...no forensic method has been rigorously shown to have the capacity to consistently, and with a high degree of certainty, demonstrate a connection between evidence and a specific individual or source", we also designed and implemented a forensic interpretation model called \emph{the CLIFF Avoidance Model}, CAM. The tools in CAM were chosen to address this issue of the lack of consistency in forensic methods. We define this lack of consistency as $brittleness$ and recommend that forensic scientists include a $brittleness$ measure for their models so that they know how reliable their models results are.

CAM successfully uses 1NN as the baseline model, and CLIFF to reduce its $brittleness$ in order to achieve the desired consistency for forensic interpretation. In other words, with CAM, for an interpretation to change, there must be sufficient evidence.

From these collective findings we conclude that CLIFF's simple linear algorithm is an excellent addition to the field of prototype learning and a worthy addition to forensic models. It performs as well as and sometimes better than some published PLS which are complex and therefore computationally expensive. Furthermore, since it is designed to choose prototypes which best represent an individual class, an overfitting bias is avoided.

\section{Future Work}

\subsection{Using CLIFF with Other Classifiers}

Although CLIFF was created to eliminate the drawbacks of nearest neighbor classifiers, the next step would be to try CLIFF with other standard classifiers particularly decision tree learners which have a tendency to produce complex theories \cite{me07}. We conjecture that reducing the size of the training set with CLIFF beforehand will produce shorter trees and therefore simpler theories.  

\subsection{Using CLIFF to Optimized Feature Subset Selection}

In Chapter \ref{chapter:forensics} we reduced the dimensionality of the data set used with FastMap \cite{fastmap}. With the new data set CLIFF was still able to produce favorable results. It would be interesting to see if the same can happen using feature subsets. 

\subsection{Comparing CAM to Other Forensic Models and Forensic Data Sets}
In this work CAM was only tested on one data set and it's performance was not compared directly to other forensic models. We intend to first test other methods for $brittleness$, second, identify any causes of $brittleness$ if present and finally compare their performance with CAM.


%In future work, we plan to test CLIFF on larger data sets. 

%The principal purpose of this work was to address the concern of the National Academy of Sciences which stated in a report \cite{09NAS} that: 

%\begin{quotation}
%With the exception of nuclear DNA analysis, however, no forensic method has been rigorously shown to have the capacity to consistently, and with a high degree of certainty, demonstrate a connection between evidence and a specific individual or source. \cite{09NAS}, p6
%\end{quotation}

%Our answer to this is, a novel approach for the evaluation of trace forensic evidence. With a data set donated by \cite{Karslake09}, made up of the infrared spectra of the clear coat layer of a range of cars, we showed that:

%\begin{itemize}
%\item CLIFF creates strong models with $low$ brittleness levels
%\item The CLIFF selector, based on a PLA, can further reduce the brittleness of a model
%\item The levels of brittleness differ significantly before and after the use of the CLIFF selector (Mann Whitney U test)
%\end{itemize}


%Although we contend that CLIFF can be applied to any type of trace evidence, in future work we hope to acquire more data sets to test CLIFF on. Also, direct comparison with other evaluation models will be investigated.