The output of all the trials will be combined to form the final results. This information is then used to generate a "abcd" stats(\fig{abcdstats}), or a confusion matrix\cite{confusion}, for each trial. One item of importance, as it deviates from the "normal" method of benchmarking classifiers, I will only report statistics on the defective modules. The correctly identified non-defective modules will be ignored.
\input{figures/confusionmatrix}
From the gathered "abcd" stats, two other statistics are generated: The Probability of Detection(PD - \eq{PD}), and the Probability of False Detection(PF - \eq{PF}).
\begin{equation}
\textit{PD} = \frac{D}{B+D} 
\label{eq:PD}
\end{equation}
\begin{equation}
\textit{PF} = \frac{C}{C+A} 
\label{eq:PF}
\end{equation}
These are the statistics used for further examination. The combined set of Probability of Detection and Probability of False alarm rate statistics for each trial is then ranked according to a Mann-Whitney-Wilcoxin\cite{MWW} U test. With the Mann-Whitney-Wilcoxon ranks, you can determine, within a certain percentage of confidence, whether two methods are equivalent, if one method out-performs the other method. I will report Mann-Whitney-Wilcoxon ranks within a 95\% confidence interval unless stated otherwise. I will also show quartile charts of the Probability of Detection and Probability of False alarm rates. The quartile charts will show the $2^{nd}$ quartile, the median value, and the $3^{rd}$ quartile.

I will compare the classifiers described in \chap{related} and \chap{clump} to each-other, showing their relative performance to the other classifiers. I will also explore the application of K-Means and Greedy Agglomerative Clustering, comparing the classifier with and without the clusterer being applied. These various experiments will paint a picture about the usefulness of locality, both in the classifier and in the application of the clusterers, being applied to software defect prediction.
