\subsection{Threats to Validity}
\label{section:ttv}

As with any empirical study, biases can affect the final results. Therefore any conclusions made from this work must be considered with the following issues in mind:

\be 

%\item The BORE algorithm in CLIFF relies on the frequency counts of attribute values to determine if a particular value is important for selecting a class. So, in a case where more instances are added to a data-set the criterion for a class can change accordingly.% The size of the data-set can determine the criteria for each class.

\item The distance metric used in this work is the simple Euclidean distance. The euclidean distance is designed for numeric attribute values with attribute of approximately equal ranges. Since all data-sets used here have been pre-processed to have discrete values the euclidean distance is applied by measuring the distance as $1$ if the values are different and $0$ if the values are the same. In this case it is difficult to judge if the ranges are approximately equal and so other distance metrics may do better than the Euclidean distance.

\item Clustering is used in the forensics. When the data-sets are unsupervised such as the forensic data-set used in the case study, k-means clustering is used to form similar groups. The number of groups is the determined by the parameter and so can influence the results of the criteria and the number of instances selected.

\item In evaluating the generalizability of our work one must consider that these results are dependent on the use of the popular UCI data-sets and the forensic data-set - cannot generalize the results. Therefore, generalizability is reduced because of the lack of random samples used.

\item For the simplicity of explaining CLIFF, we used k=1 in KNN. However the performance of the ISM and KNN can be improved or degraded according to the value of k. The effects of k should be taken into account in future work. 

\item The field of instance selection is active, and so it would be difficult compare the performance of CLIFF to all of them. All we can do is choose a subset of ISM from the field and fully describe our experimental procedure so that others can perform further evaluations.

\ee

\section{Conclusions and Future Work}
\label{section:conclusion}

\subsection{Conclusions}
%Our goal in this work has been to evaluate the hypothesis that \emph{there are critical values for each attribute of a data-set which can be used to select instances from the data-set in order to improve the performance of nearest neighbor classifiers}. 

This work has compared a novel instance selection method called CLIFF with three(3) other ISM. CLIFF uses a ranking algorithm to create a criterion for each class in a data-set. These criteria are then used to select instances. CLIFF was then tested on nine(9) data-sets and compared with three(3) ISM (CNN, MCS and PSC). We showed that CLIFF:

\be
\item has a time complexity of $O(n)$;
\item reduces training sets to a range of 2 to 47\%;
\item has $acc$, $pd$, $prec$ and $pf$ results which compares favorably with 1NN and other ISM in several standard data-sets;
\item does not significantly increase the number of instances selected in the presence of noise as compared with other ISM;
\item reduces $brittleness$ substantially in most data-sets used.
\ee

Having analysed CLIFF, we then applied it to a real-world forensic data-set. We designed and implemented a forensic interpretation model called \emph{the CLIFF Avoidance Model}, CAM. The tools in CAM were chosen to address this issue of the lack of consistency in forensic methods. We define this lack of consistency as $brittleness$ and recommend that forensic scientists include a $brittleness$ measure for their models so that they know how reliable their models results are.

%CAM successfully uses 1NN as the baseline model, and tried CLIFF to reduce its $brittleness$ in order to achieve the desired consistency for forensic interpretation. In other words, with CAM, for an interpretation to change, there must be sufficient evidence.

%From these collective findings 
We conclude that CLIFF's simple linear algorithm is an excellent addition to the field of instane seletion and a worthy addition to forensic models. It performs as well as and sometimes better than some published ISM which are complex and therefore computationally expensive. Furthermore, since it is designed to choose instances which best represent an individual class, and overfitting bias is avoided.

\subsection{Future Work}

\subsubsection{Using CLIFF with Other Classifiers}

Although CLIFF was created to eliminate the drawbacks of nearest neighbor classifiers, the next step would be to try CLIFF with other standard classifiers particularly decision tree learners which have a tendency to produce complex theories \cite{me07}. We conjecture that reducing the size of the training set with CLIFF beforehand will produce shorter trees and therefore simpler theories.  

\subsubsection{Using CLIFF to Optimized Feature Subset Selection}

In Section \ref{section:forensics} we reduced the dimensionality of the data-set used with FastMap \cite{fastmap}. With the new data-set CLIFF was still able to produce favorable results. It would be interesting to see if the same can happen using feature subsets. 

\subsubsection{Comparing CAM to Other Forensic Models and Forensic data-sets}
In this work CAM was only tested on one data-set and it's performance was not compared directly to other forensic models. We intend to first test other methods for $brittleness$, second, identify any causes of $brittleness$ if present and finally compare their performance with CAM.
