\subsection{Contrast Set Learning}
Much learning in life is done through observations of examples. If one considers various similar examples to be grouped into separate populations, much of what can be learned lies in what separates them. Contrast set learning is a quality optimization technique that provides the ability to distinguish the differences between two populations. These differences are where we draw insight from with regard to why the two populations are unlike each other. In the case of {\W}, the sets to be contrasted are the two sets of similar cases which exemplify the defective and flawless historical instances. We contrast the two sets by examining the frequency with which each metric appears in the two sets. What this gives us is the ability to identify specific metrics which can be correlated to an instance's membership within the flawless, or defective, set. In essence, {\W} was designed as an instance based reasoner with the added ability of performing contrast set learning.

\subsection{Which Algorithm}
{\W} is an instance based reasoner written in lisp, designed to discover rules that predict for given classes. To perform its function, {\W} requires two sets of data: a training dataset and a testing dataset. The general operation of {\W} is described in the following steps:
\bi
\item
The data is first discretized from continuous metrics into "n" bins of equal width. For the purpose of our research we used n=2,3.
\item
{\W} maintains a stack of sorted metric combinations we refer to as rules. These combinations are sorted by a user-customizable search bias which in our case was how well they predicted for historical instances with defects.
\item
At the beginning of a run, {\W} contrasts the defective and flawless sets of instances from the training dataset and derives simple rules that get pushed onto its stack.
\item
{\W} randomly picks two rules from its stack. The rules ranked higher by our search bias are preferred. 
\item
The stack entries picked in the previous step are combined into a new rule.
\item
The new rule is scored and subsequently injected into the sorted stack of previous rules.
\item
{\W} goes back to the stack, picks two more random rules and continues on and on.
\item
This procedure continues until there is no more noticeable improvement in prediction or until the maximum number of lives is reached. This is a  user-definable variable for which we used a value of 5.
\item
Once either of the previous conditions happens, {\W} returns the highest ranked predictor found in its stack. We then apply this rule to the testing dataset and thus compute the probability of detecting real faults, (PD), and the probability of returning false positives, (PF). This result validation step helps show that the defect predictors obtained from {\W} are sane.
\ei

\subsection{Datasets}
The thirteen datasets we used contain quality data for several open source projects and are publicly available at promisedata.org. These datasets have been used for defect prediction research before by Jureczko and Madeyski\cite{Jureczko}. Each dataset was originally split between several individual files, some only consisted of two files whereas others were made up of four or five files. Each file within a dataset contains data pertaining to a particular version of the project the dataset represents. Naturally these datasets are of various sizes, and, as mentioned above, contain data from different numbers of project versions. Some preprocessing was required to use these datasets. Before being able to run them through {\W}, they were all converted from their original layout to a {\W}-friendly lisp-based format. The names of the datasets we used are as follows: ant, camel, forrest, jedit, log4j, lucene, pbeans, poi, prop, synapse, velocity, xalan, and xerces.

\subsection{Testing Methodology}
Since our datasets were already organized in a neat "project version per file" fashion, they lent themselves easily to training and testing on their different versions, that is testing whether or not the same defect predictor surfaced when training and testing on the multiple project versions contained within each dataset. This concept is referred to as intra-project stability. To show this intra-project predictor stability {\W} would ideally output the same defect predictor rule for all, or at least a majority of, train/test cycles performed over a dataset's different versions.

That being said, we decided that simply training on one version and testing on another would be unacceptable since the datasets each contain a limited number of versions. In order to increase the amount of usable training data at our disposal we concatenated the individual files and used these to train before testing on the original files. If for example a dataset, called ds, is comprised of four files: ds-a, ds-b, ds-c, and ds-d, all of which represent different versions of the same software project, then the files ds-ab, and ds-abc, are concatenations of the respective initial files (ds-ab is a concatenation of the contents of ds-a and ds-b and so on). We would then perform training on ds-a and testing on ds-b, training on ds-ab and testing on ds-c, training on ds-abc and testing on ds-d. In the end, ideally we would have all three train/test cycles result in the same predictor. Besides the predictor rule we expect to get at the end of a train/test cycle, we also compute probability values for predicting real defects, (PD), as well as for returning false positives, (PF). The reason behind including these probability calculations is the understanding that a high value for PD observed alongside a low value for PF indicates our predictor is in fact predicting correctly.






