%notes - instead of reduction % use storage requirements
% find the most popular of the IS and use those in the table


\section{CLIFF Assessment}
\label{section:assess}
In Section \ref{section:cliff}, we discussed the design and operation of CLIFF with the help of a simple example. We also showed its time complexity as linear - $O(n)$ in Section \ref{section:time}. In this section, we look at how CLIFF performs against other IS with three(3) evaluation methods. CLIFF is evaluated by first comparing its performance with three(3) IS spanning the decades from 1968 (Hart's CNN \cite{Hart68a}) to MCS in 1994 \cite{Dasarathy94} and finally 2010 PSC \cite{lot2010}. We then move on to examine the noise tolerance of each IS studied here by introducing artificial noise to the training sets. Finally we take a look at what we call the $brittleness$ measure. Brittleness, is defined as \emph{a tiny change in the input data can lead to a major change in the output}. As mentioned previously, we view instance selection as a viable method to decrease $brittleness$ and the following sections will show that CLIFF does a better job of reducing the impact of $brittleness$ than other IS evaluated in this work.

%\section{CLIFF Assessment on Standard Data Sets}
%\label{section:assess}
\subsection{Data Sets}
\fig{info} lists the nine(9) data sets used to assess CLIFF. The number of instances and attributes per instance are shown for each data set, along with the number of distinct classes of instances. All of these data sets were acquired from the UCI repository \cite{Frank+Asuncion:2010}. Since CLIFF is designed for data sets with discrete values, all data sets with numeric attribute values are discretized by the Fayyad and Irani \cite{Fayyad1992} discretizer implemented in WEKA. 

\begin{figure}
\begin{center}
\scalebox{0.8}{
\begin{tabular}{ | l | l | l | l |}
\hline
Data & Instances & Attributes & Class \\ \hline
Breast Cancer (bcancer) & 286 & 9 & 2 \\ \hline
Ecoli (ec) & 366 & 8 & 8 \\ \hline
Glass (gl) & 214 & 9 & 6 \\ \hline
Heart (ht) & 303 & 13 & 2 \\ \hline
Hepatitis (hep) & 155 & 19 & 2 \\ \hline
Iris (ir) & 150 & 4 & 3 \\ \hline
Labor (lb) & 57 & 16 & 2 \\ \hline
Lymph (ly)& 148 & 18 & 4 \\ \hline
Vote (vt) & 435 & 16 & 2 \\ \hline
\end{tabular}}
\end{center}
\caption{Data Set Characteristics}
\label{fig:info}
\end{figure}     

\subsection{Experimental Method}
\label{section:brit}

We evaluate CLIFF as an instance selector on standard data sets in cross validation experiments. Its performance compared with KNN, CNN, MCS and PSC is characterized using five(5) measures of evaluation:
\be
\item using probability of detection (pd);
\item precision (prec); 
\item probability of false alarm (pf);
\item brittleness (brit);
\item reduction percent (red) which can indicate noise tolerance.
\ee

One(1) to three(3) above is completed as follows \cite{burak}: By allowing A, B, C and D to represent true negatives, false negatives, false positives and true positives respectfully, it then follows that \emph{pd} also known as recall, is the result of true positives divided by the sum of false negative and true positives \emph{D / (B + D)}. While pf is the result of: \emph{C / (A + C)}. Finally, precision is the result of: \emph{D / (C + D)}. The $pd$, $prec$ and $pf$ values range from 0 to 1. When there are no false alarms $pf$ = 0 and at 100\% detection, $pd$ = 1.

Next, brittleness is measured by distance each test instance moves before changing its original predicted target class. Intuitively, the further away the test instance has to move the less $brittle$ the model. Collecting the distances of each test instance with a predicted target class to the nearest training instance with a different target class. This is done for all IS and knn where $k=1$. The distances generated are joined, sorted and labeled according to their position in the list. For example, let us say that A and B are IS with the distance values of \{2, 2, 2, 3, 4, 3, 77\} and \{6, 7, 3, 9, 1, 1, 1, 100\} respectively. After being joined, sorted and labeled the result is as follows:

\begin{figure}[ht!]
\scalebox{0.7}{
\begin{tabular}{l*{14}{c}r}
IS  & B & B & B & A  & A & A & A & A & B &  A &  B &  B &  B &  A  &  B \\
\hline
Sort & 1 & 1 & 1 & 2 & 2 & 2 & 3 & 3 & 3 &  4 &  6 &  7 &  9 & 77 & 100  \\
Position & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 & 11 & 12 & 13 & 14 &  15  \\
Label & 1 & 1 & 1 & 5 & 5 & 5 & 8 & 8 & 8 & 10 & 11 & 12 & 13 & 14  & 15  \\
\end{tabular}}
\end{figure}
	
As shown, labels are assigned according to the position of a value in the list, however, if values are the same, the mean of their position values are used as a label.

Last, the reduction percent is simply the number of the instances selected as compared to the original training data. This measure is particularly important in terms of noise. According to \cite{wilson00}, in the presence of class noise very few instances will be removed from the training set thus increasing the reduction percent. Using the class noise injector described in \cite{wilson00}, noise is introduced to training data by randomly changing the target class of \{10, 20, 30, 40\}\% of the instances in the training data to any other target class value.

Our experiment design follows the pseudo code given in \fig{knnexp1} for the standard data sets in \fig{info}. For each data-set, tests were built from 20\% of the data, selected at random. Instances were then selected by an instance selector from the remaining 80\% of the data. Next, models were learned from the selected instances using the {\em 1 nearest neighbor rule} and the euclidean distance as the distance metric.

This procedure was repeated 5 times, randomizing the order of data in each data-set each time. In the end CLIFF is tested and trained 25 times for each data-set. 

\begin{figure}[ht!]
\renewcommand{\baselinestretch}{1}
\begin{algorithmic} \footnotesize
\STATE {\em data = [breastcaner ecoli glass heart hepatitis iris labor lymph vote]}
\STATE {\em learner = [knn]}
\STATE {\em IS = [knn cliff cnn mcs psc]}
\STATE {\em noise = [10 20 30 40]}
\STATE {\em brit = list of distances: \{D(predicted test) - D(NUN)\}}
\STATE {\em red = reduction percentage}
%\STATE {\em $stat_{test}$ = [Mann Whitney]}
\STATE {\em count = 0}

\FORALL {~IS}
  \REPEAT
    \FORALL {~d in data}
     \FORALL {~n in noise}
     \STATE train = random 80\% of data
     \STATE test = d - train
		
     \COMMENT {Construct model from train data}
     \STATE $r_{train}$ = Reduce train with IS
     \STATE model = Train learner with $r_{train}$
     
     \COMMENT {Evaluate model on test data}
     \STATE [pd, prec, pf, red, brit] = model on test
   \ENDFOR
   \ENDFOR
   \STATE {\em count = count + 1}
  \UNTIL {\em count = 5}
\ENDFOR  	
\end{algorithmic}
\caption{Pseudo code for Experiment}
\label{fig:knnexp1}
\end{figure}

\begin{figure*}
  \centering
  \subfloat{\label{}\includegraphics[width=0.195\textwidth]{red0}}                
  \subfloat{\label{}\includegraphics[width=0.195\textwidth]{red1}}
  \subfloat{\label{}\includegraphics[width=0.195\textwidth]{red2}}
  \subfloat{\label{}\includegraphics[width=0.195\textwidth]{red3}}
  \subfloat{\label{}\includegraphics[width=0.195\textwidth]{red4}}
  \caption{Effect of noise on reduction percent. As noise increases from 0 - 40\%, inferior instance selectors generally increase in reduction \%, i.e. the number of instances selected from each training set increases as noise increases.}
  \label{fig:red-results}
\end{figure*}

\begin{figure}[ht!]
\begin{center}
\includegraphics[scale=1]{breastcancer}
\end{center}
\caption{Visual illustration of brittleness reduction in the breast cancer data-set for KNN and the IS. Here, the greater the position of each test instance with a predicted target class from an instance with a different target class the better.}\label{fig:brit-results}
\end{figure}


\begin{comment}
\begin{figure*}
  \centering
  \subfloat{\label{}\includegraphics[width=0.3\textwidth]{breastcancer}}                
  \subfloat{\label{}\includegraphics[width=0.3\textwidth]{ecl}}
  \subfloat{\label{}\includegraphics[width=0.3\textwidth]{glass}} \\
  \subfloat{\label{}\includegraphics[width=0.3\textwidth]{hc}}
  \subfloat{\label{}\includegraphics[width=0.3\textwidth]{hepatitis}}
  \subfloat{\label{}\includegraphics[width=0.3\textwidth]{iris}} \\
  \subfloat{\label{}\includegraphics[width=0.3\textwidth]{labor}}
  \subfloat{\label{}\includegraphics[width=0.3\textwidth]{lymph}}
  \subfloat{\label{}\includegraphics[width=0.3\textwidth]{vote}}

  \caption{Visual illustration of brittleness}
  \label{fig:brit-results}
\end{figure*}




%The results were visualized using \emph{quartile charts} as in \cite{burak}. To generate these charts the performance measures for the $pd$s and $pf$s are sorted to get the median, lower and upper quartile of numbers. For our quartile charts, the upper and lower quartiles are marked with black lines; the median is marked with a black dot; and the vertical bars are added to mark the 50\% percentile value. \fig{qc} shows an example where the upper and lower quartiles are 39\% and 59\% respectively, while the median is 49\%. 

\begin{figure}[ht!]
  \begin{center}
  \scalebox{0.5}{
    \begin{tabular}{l}
      \resizebox{100mm}{!}{\includegraphics{quartc}}      
    \end{tabular}}
    \caption{}
    \label{fig:qc}
  \end{center}
\end{figure}
\end{comment}

%Finally, the Mann-Whitney U test was used to test for statistical difference of the different IS. These results are shown as rank values starting at one(1). The lower the rank value, the better the performance of the instance selector. Please note that IS with the same rank value, are not statistically different. The following sections describes the experiments and discusses the results.

%\subsection{Is CLIFF viable as an IS for NN?}

\subsection{Experimental Results}

To answer the question: Is CLIFF viable as an instance selector for NN? We first look at the results for reduction percent. Second, we provide a visual illustration regarding the reduction of brittleness by each instance selector studied. Third, look at the effects of noise on the performance (pd, prec and pf) of KNN and CLIFF. Finally, we determine if CLIFF is statistically better than KNN and the 3 IS studied here.


\subsubsection{Reduction Percent}
This measure addresses the issue of large storage requirement for the NN algorithm. Additionally, it also gives an indication of the noise tolerance of different IS and NN.

\fig{red-results}, visually illustrates the reduction percent results for each data-set. Each chart reports at different noise levels from zero to 40\%. When there is no noise present, all the IS reduces the training set to at most 60\%. CLIFF is impressive, with its reduction percent at 5\% and below for all data-sets except for Ecoli where it got as high as 47\%. As noise increases CLIFF reports generally stable results while the other instance selectors show a gradual increase in their reduction size. Therefore, in addition to a small storage requirement, these results clearly indicate that unlike the other IS, CLIFF is resistant to noise.


\subsubsection{Visualization of Brittleness}
Recall our contention that IS are a feasible way to reduce brittleness in NN. In \fig{brit-results} the pattern is very clear. CLIFF does a much better job of reducing $brittleness$ in all cases than any of the other IS. \fig{brit-results} represents the result for the breast-cancer data-set used in this work. This result is generally replicated in the other eight data-sets used in this work. They all show that the CLIFF test instances have to move further away (most of the time) before there is a change in their target classes. These results are confirmed by a Mann Whitney statistical test, which reveal that the CLIFF brittleness results are statistically different and better than those of the other IS. 

\subsubsection{The Effects of Noise}

The goal here is to see if the CLIFF works well in the presence of noise. This is important because according to \cite{wilson00}, the performance of an instance selector can suffer especially if noisy instances are kept while good instances are removed. In \fig{noise} we compare the performance (pd, prec and pf) of CLIFF to KNN. In the presence of noise we expect CLIFF performance measures to remain study while those of KNN degrade significantly. The results in \fig{noise} corroborates this intuition. For KNN as noise increases both the $pd$ and $prec$ measures decrease. This is especially true for the Ecoli, Glass and Iris data-sets whose measures show a drastic reduction when 20\% class noise is introduced. On the other hand, the CLIFF measures tend to remain steady as noise increases. The exceptions to this are the Hepatitis, Labor and Lymph data-sets. 

\subsubsection{Is CLIFF significantly better than other IS?}

\fig{better1} to \fig{better3} summarizes the results of this experiment. Here, the $pd$, $prec$ and $pf$ measures of KNN, MCS and PSC are subtracted from the corresponding CLIFF measures showing whether or not their results are better the CLIFF. Note that missing values indicate that the result was not significantly different to CLIFF according to the Mann-Whitney U test. The $pd$ measures in \fig{better1} and the $prec$ measures in \fig{better2}, show CLIFF outperforming all other IS and KNN for all data-sets except for Labor when no noise is included. Similarly, in \fig{better3}, CLIFF shows its superiority of all other IS for $pf$ measures, except in the case of Labor at 0\% noise.



%The goal here is to see if the performance of CLIFF is comparable or better than the plain k nearest neighbor (KNN) algorithm, and the CNN, MCS and PSC instance selectors. 

%In this experiment we compare the performance of predicting the target class using the entire training set to using only the instances selected by the IS including CLIFF. To accomplish this, our experiment design follows the pseudo code given in \fig{knnexp1} for the standard data sets. For each data set, tests were built from 20\% of the data, selected at random. Instances are then selected by an instance selector from the remaining 80\% of the data. Next, models were learned from the selected instances.

%This procedure was repeated 5 times, randomizing the order of data in each data-set each time. In the end CLIFF is tested and trained 25 times for each data set. 



%\subsubsection{Results from Experiment \#1}
%The results for this experiment and Experiment 2 (discussed in the following section) are shown in \fig{res1}. For each data-set, the first column shows results where no noise is added to the training set. the each figure shows the results for a $clean$ data-set (without noise) and a $noisy$ data-set (with noise). For both the clean and noisy data-sets, the [pd, pf] quartile charts, the percentage of the data-set used for training after using the IS and the rank values showing the significant difference between the IS and KNN are presented. 

%Let us focus on the $clean$ results for each data-set from \fig{info}. First, we compare the CLIFF results with those of the baseline KNN. As shown, despite using only 9\% to 15\% of the training set, the pds and pfs results of CLIFF compares favorably with those of KNN showing similar or better rank values in most cases. For example, the Mammography(mm) data set (\fig{results106}) ranks as number one(1) for both the pd and pf results while KNN ranks at number three(3). In \fig{results105}, the Liver(lv) data set exhibits statistically similar results for pd while for pf CLIFF has a much better statistical performance than KNN. \fig{results101} to \fig{results104} also show encouraging results for CLIFF as compared with KNN, however \fig{results100}, the Breast Cancer(bc) data set shows an exception, with the pd and pf statistical results for KNN better than CLIFF.

%Next, let us consider the results of the IS. As compared with the other IS, CLIFF does as well as or better than the others in all but one(1) case. However, although for the $bc$ data set, CNN present statistically better pd and pf results, it does so with 62\% of the training data for median pd and pf values of 55\% and 39\% respectively, while CLIFF only needed 11\% of the training data for median pd and pf values of 67\% and 20\% respectively.   

%So generally, CLIFF has markedly lower pfs than the other IS and KNN, this is especially true for the $hh$ and $ir$ data sets. Also the low pfs does not come at the cost of lower pds, infact, CLIFF's median pd results are competitive (sometimes even the best) of all the other IS. 

\begin{comment}
\begin{figure}
\begin{center}
\scalebox{0.9}{
\begin{tabular}{p{2cm} | l | r | r | r}
\hline
UCI Data  & IS & pd & prec & pf \\
\hline
\multirow{5}{BreastCancer} & knn & 62.5 & 66.7 & 35.1 \\
  & knn+cliff & 58.3 & 66.0 & 35.5 \\
  & knn+cnn & 63.2 & 63.6 & 35.0 \\
  & knn+mcs & 52.9 & 52.4 & 46.7 \\ 
  & knn+psc & 50.0 & 48.0 & 48.8 \\ \hline
\multirow{5}{Ecoli} & knn & 62.5 & 66.7 & 1.6  \\
  & knn+cliff & 57.1 & 75.0 & 1.7 \\
  & knn+cnn & 33.3 & 37.5 & 1.7 \\
  & knn+mcs & 33.3 & 31.2 & 1.6 \\ 
  & knn+psc & 20.0 & 26.9 & 3.0  \\ \hline
\multirow{5}{Glass} & knn & 66.7 &71.4& 2.6 \\
  & knn+cliff & 60.0 &83.3 &2.6 \\
  & knn+cnn & 50.0 &52.9 &4.0 \\
  & knn+mcs & 50.0 &50.0 &5.0 \\ 
  & knn+psc & 35.7 &33.0& 8.8  \\ \hline
\multirow{5}{HeartC} & knn & 76.7 &75.0& 23.1  \\
  & knn+cliff & 82.6& 83.3& 17.2 \\
  & knn+cnn & 74.2 &75.0 &25.7 \\
  & knn+mcs & 54.1 &53.3& 43.8 \\ 
  & knn+psc & 50.0 &50.0& 50.0  \\ \hline
\multirow{5}{Hepatitis} & knn & 83.3& 82.6 &16.7  \\
  & knn+cliff & 81.8& 77.8 &17.4 \\
  & knn+cnn & 77.8 &71.4& 21.7 \\
  & knn+mcs & 63.6 &68.8& 34.6 \\ 
  & knn+psc & 60.9 &66.7& 38.5  \\ \hline
\multirow{5}{Iris} & knn & 100.0 & 100.0 & 0.0  \\
  & knn+cliff & 100.0 & 100.0 & 0.0 \\
  & knn+cnn & 90.9& 88.9& 4.3 \\
  & knn+mcs & 90.9 &90.0& 4.5 \\ 
  & knn+psc & 90.9 &92.3 &4.3 \\ \hline
\multirow{5}{Labor} & knn & 100.0 & 100.0 & 0.0  \\
  & knn+cliff & 88.9 &88.9& 11.1 \\
  & knn+cnn & 90.9 &85.7& 0.0 \\
  & knn+mcs & 100.0 & 100.0 & 0.0 \\ 
  & knn+psc & 100.0 & 100.0 & 0.0  \\ \hline
\multirow{5}{Lymph} & knn & 78.6& 78.9& 0.0  \\
  & knn+cliff & 76.9& 70.0& 6.7 \\
  & knn+cnn & 66.7 &66.7& 0.0 \\
  & knn+mcs & 68.8 &70.6& 0.0 \\ 
  & knn+psc & 58.8 &63.6& 3.4  \\ \hline
\multirow{5}{Vote} & knn & 92.1& 92.2& 7.7  \\
  & knn+cliff & 89.8& 88.1& 9.8 \\
  & knn+cnn & 88.9 &88.2 &11.1 \\
  & knn+mcs & 70.4 &72.9& 29.3 \\ 
  & knn+psc & 72.2 &71.4& 27.0  \\ \hline
\end{tabular}}
\caption{pd, prec and pf results for Experiment \#1}
\label{fig:res1}
\end{center}
\end{figure}
\end{comment}

%\subsection{Is CLIFF more tolerant to noise than other IS?}



%With that said, in this experiment we repeat Experiment \#1, only this time noise is introduced to training data by randomly changing the target class of 10\% of the instances in the training data to any other target class value. The the results here will indicate how well the 1NN classifier along with the different IS are able to predict the correct target class even if some of its training data is faulty \cite{wilson00}. 

%\subsection{Results from Experiment \#2}
%The $noisy$ tables in \fig{noise} displays the results of Experiment \#2. The first thing to recognize is that compared to the $clean$ tables, there is a general degradation of the pd and pf results for each data set. However in some cases such as the pd results for $bc$ the pds can degrade by as little as 1\%. The second thing to recognize is that while the $sizes$ of the training data basically remains the same for CLIFF (differences of no more that 2\%) for all data sets, the other IS display a general increase in the training set sizes except for PSC whose training size decrease from 42\% to 40\%. 
\begin{comment}
\begin{figure*}[!ht]
\begin{center}
\renewcommand{\baselinestretch}{0.7}
\begin{tabular}{p{0.25in}@{}c@{}c}
&KNN & CLIFF \\ \hline
pd&\includegraphics[width=2.5in]{knnpd.pdf} &
\includegraphics[width=2.5in]{cliffpd.pdf}
\\\hline
prec&\includegraphics[width=2.5in]{knnprec.pdf} &
\includegraphics[width=2.5in]{cliffprec.pdf}
\\\hline
pf&\includegraphics[width=2.5in]{knnpf.pdf} &
\includegraphics[width=2.5in]{cliffpf.pdf}
\\
\end{tabular}
\end{center}
\caption{Effects of noise.}
\label{fig:noise}
\end{figure*}
\end{comment}
%%%%%%%%%%%%%

\begin{figure*}[!ht]
\begin{center}
\renewcommand{\baselinestretch}{0.5}
\begin{tabular}{p{0.20in}@{}c@{}c@{}c}
&pd & prec & pf \\ \hline
KNN&\includegraphics[width=2in]{knnpd.pdf} &
\includegraphics[width=2in]{knnprec.pdf} &
\includegraphics[width=2in]{knnpf.pdf} &
\\\hline
CLIFF&\includegraphics[width=2in]{cliffpd.pdf} &
\includegraphics[width=2in]{cliffprec.pdf} &
\includegraphics[width=2in]{cliffpf.pdf}
\\
\end{tabular}
\end{center}
\caption{Effects of noise: Performance measures (pd, prec and pf) of CLIFF vs. KNN in the presence of noise.}
\label{fig:noise}
\end{figure*}

%%%%%%%%%%%%%

\begin{figure}[!ht]
\begin{center}
\renewcommand{\baselinestretch}{0.6}
{\scriptsize
\begin{tabular}{lr|rrrrr}
& & \multicolumn{5}{c}{percent noise}\\
& & 0 & 10 & 20 & 30 & 40\\\hline
\input{pds}
\end{tabular}
}
\caption{PDS: other - (knn+cliff). Note: {\em negative} values means CLIFF is performing {\em better}.} 
\label{fig:better1}
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
\renewcommand{\baselinestretch}{0.6}
{\scriptsize
\begin{tabular}{lr|rrrrr}
& & \multicolumn{5}{c}{percent noise}\\
& & 0 & 10 & 20 & 30 & 40\\\hline
\input{precs}
\end{tabular}
}
\caption{PRECISION: other - (knn+cliff). Note: {\em negative} values means CLIFF is performing {\em better}.} 
\label{fig:better2}
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
\renewcommand{\baselinestretch}{0.6}
{\scriptsize
\begin{tabular}{lr|rrrrr}
& & \multicolumn{5}{c}{percent noise}\\
& & 0 & 10 & 20 & 30 & 40\\\hline
\input{pfs}
\end{tabular}
}
\caption{PFs: other - (knn+cliff). Note: {\em positive} values means CLIFF is performing {\em better}.} 
\label{fig:better3}
\end{center}
\end{figure}
\begin{comment}
\subsection{Is CLIFF better at reducing brittleness than other IS?}

In this work, $brittleness$ refers to the following:

\begin{quote}
\emph{Brittleness} is a measure of whether a solution (predicted target class) comes from a region of similar solutions or from a region of dissimilar solutions. Or, looking at this another way, how far would a test instance have to move before a different target class is predicted.
\end{quote}


\begin{figure}[ht!]
  \begin{center}
  \scalebox{1}{
    \begin{tabular}{l}
      \resizebox{100mm}{!}{\includegraphics{ir2fss}} \\
      \resizebox{100mm}{!}{\includegraphics{ir2fssaf}}      
    \end{tabular}}
    \caption{}
    \label{fig:charts2}
  \end{center}
\end{figure} 

Take for example the \emph{Before CLIFF} chart in \fig{charts2}, the classes $versicolor$ and $virginica$ obviously show severe overlap. Also, the versicolor test instance represented by the purple square does not have to move very far before it can change its predicted target class to $virginica$. Looking now at the \emph{After CLIFF} chart in \fig{charts2}, after applying CLIFF, a subset of instances are selected as prototypes. This increases the distance from the $versicolor$ test instance to a prototype with the $virginica$ class thereby reducing $brittleness$.

With this measure in mind, the goal here is to see how each prototype learning scheme studied here reduces the brittleness of the KNN model where k=1. Brittleness will be measured by distance each test instance moves before changing its original predicted target class. Intuitively, the further away the test instance has to move the less $brittle$ the model. The experiment design for $brittleness$ can be done in conjunction with Experiment 1 by collecting the distances of each test instance with a predicted target class to the nearest training instance with a different target class. This is done for each prototype learning scheme and 1NN. The distances generated are joined, sorted and labelled according to their position in the list. for example, let us say that A and B are IS with the distance values of [2, 2, 2, 3, 4, 3, 77] and [6, 7, 3, 9, 1, 1, 1, 100] respectively. After being joined, sorted and labelled the result is as follows:

\begin{figure}[ht!]
\scalebox{0.7}{
\begin{tabular}{l*{14}{c}r}
IS  & B & B & B & A  & A & A & A & A & B &  A &  B &  B &  B &  A  &  B \\
\hline
Sort & 1 & 1 & 1 & 2 & 2 & 2 & 3 & 3 & 3 &  4 &  6 &  7 &  9 & 77 & 100  \\
Position & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 & 11 & 12 & 13 & 14 &  15  \\
Label & 1 & 1 & 1 & 5 & 5 & 5 & 8 & 8 & 8 & 10 & 11 & 12 & 13 & 14  & 15  \\
\end{tabular}}
\end{figure}
	
As shown, labels are assigned according to the position of a value in the list, however, if values are the same, the mean of their position values are used as a label.

\subsection{Results from Experiment \#3}
\fig{charts1} and \fig{instances} present the results for this experiment. The pattern is very clear: CLIFF does a much better job of reducing $brittleness$ in all cases than any of the other IS. Each chart in \fig{charts1} represents results for the different data sets used in this work. They all show that the CLIFF test instances have to move further away (most of the time) before there is a change in their target classes. These results are confirmed by a Mann Whitney statistical test (\fig{instances}), which show that the CLIFF results are statistically different and better than the other IS. 
\end{comment}


\subsection{Summary}
Collectively, the results of the above experiment indicates that CLIFF may be effective in the field of forensic interpretation where a very low false alarm rate $pf$ is desired. Here, CLIFF can be used to help lower the $pf$s of a forensic interpretation model. The results shown in \fig{noise} suggest this possibility in the median $pf$ results where CLIFF's values ranges from 0 to 47\% and are lower or the same as the baseline (KNN) $pf$ results.

\begin{comment}
\begin{figure*}[ht!]
%  \begin{center}
  \scalebox{0.87}{
    \begin{tabular}{l}
      \resizebox{90mm}{!}{\includegraphics{bc1}}
      \resizebox{90mm}{!}{\includegraphics{dm1}} \\
      \resizebox{90mm}{!}{\includegraphics{hc1}} 
      \resizebox{90mm}{!}{\includegraphics{hh1}} \\
      \resizebox{90mm}{!}{\includegraphics{ir1}} 
      \resizebox{90mm}{!}{\includegraphics{lv1}} \\
      \resizebox{90mm}{!}{\includegraphics{mm1}} \\
    \end{tabular}}
    \caption{Position of distance values for IS}
    \label{fig:charts1}
 % \end{center}
\end{figure*}
\end{comment}
