\section{Summary}

We have investigated the heuristics of an active learning technique and hypothesized about it's application to complex domains with much focus on software engineering.

\textit{Feature subset selection} provides us with the means to reduce features in a data set to those that are most interesting to the data set.  Principle Component Analysis is a feature subset selection algorithm that uses Eigen-Vectors to find the most significant features in the data.  We also present the Term Frequency-Inverse Document Frequency algorithm.  The algorithm counts the frequency of a term in a document and scales by the inverse term frequency to produce a weight describing the importance of a term within a document relevent to the entire collection of documents. 

\textit{Sampling methods} are very important to selecting data points.  Random sampling is technique that heavily favors exploration and will label points that will add little information to the active learner.  We look at techniques that favor exploitation to query points that are in the \textbf{region of uncertainty}.  
Hierarchical sampling applies hierarchical clustering to create a tree structure and applies statistical methods to various pruning of the trees to judge if a tree is pure (containing one class). 
Greedy sampling applies Vapnik-Chervonenkis theory to choose the most attractive points that are almost equally labeled with positive and negative labels.  
$A^2$ agnostic sampling algorithm provides a solution to noise in the data by ensuring that a hypothesis will not be removed from the hypothesis space because of a single point.  
Query by Committee uses a committee of simple prediction methods to collectively guess the label of a point and send the point to an expert if they disagree on the label.  
Finally, Warmuth suggests iterative approaches to selective sampling using nearest neighbor, support vector machine, and voted perceptron techniques and found that all three work similarly well with voted perceptron providing slightly less variance in true and false positives than the other two techniques.

\textit{Clustering methods} allow the active learner to identify areas of interest to further promote exploration and exploitation of the data set.  K-means is an general use O(kN) clustering algorithm that updates the mean point of clusters as the cluster membership changes. 
GenIc is a clustering algorithm that computes the "fitness" of a cluster to determine if the cluster survives to the next generation.  GenIc is designed for streaming data and assumes that data can only be analyzed once; GenIc boosts a O(n) complexity to facilitate this streaming data. Canopy is a clustering technique that introduces "canopies" to ensure that traditional clustering techniques only consider points within a distance threshold.  
Canopy considerably reduces the complexity of clustering algorithms; For example, K-means with canopies requires $O(\frac{n*k*f^2}{c})$ where $f$ is usually much less than $c$.  Compass is a simple O(3LN) algorithm that works by finding two points of maximum distance from each other and constructs a tree by separating data points between the two points based on a distance measure.  The algorithm then recurses on the both data sets until the variance between layers is less (uninteresting) or greater (too chaotic) than a variance threshold.  
We present results that suggest that simple feature subset selection and clustering methods perform as well as or better than more complicated methods.  We also present evidence that the Compass clustering algorithm performs considerably better than K-means and Bisecting K-means for various $k$ in several datasets from the PROMISE data repository.

We rely on \textit{Domain Experts} to label interesting data points.  Human experts are scarce and are prone to error when analyzing large batch sets of data (such as software engineering data).  We introduce the possibility of using a machine learning algorithm  to stand in for a human domain expert and establish a method for experts to communicate their opinions on their domain and terminology with one another.  
We introduce C4.5 and Naive Bayes machine learning algorithms as possible candidates for experts.  C4.5 is a decision tree learning algorithm that trains by continuously adds portions of a training set until the entire training set can be correctly classified.  Naives Bayes is a statistical learning algorithm based on Bayes' theorem.   We present results by Domingos that suggests that Naive Bayes outperforms other learners such as C4.5 in practice.