This chapter will discuss the different performancs experiments that were conducted for this thesis.  There are a total of six different
experiments that cover 4 different categories of learning that we tested the Which learner against.  Category I is a comparison to a
learner that has the same overall goal of Which, but the processes used to get the result are different.  We compare the abilities of TAR3,
discussed in \sect{tar3}, with the abilities of Which on some class UCI \cite{uci} data.  Category II is one experiment to illustrate
the limitations of our approach currently.  This experiment goes into detail about the properties of Which's rule creations and the limitations
of Which in more$-$than$-$two class data sets.  Category III is the largest and coveres three seperate experiments.  This category compares
Which to the classic learners of j48, Ripper, and \nbc in the field of defect detection in software projects.  These three experiments use
very diverse data sets that come from completely different areas of software programming.  Category IV is one experiment that
uses the results of Category III coupled with a sampling technique described in \sect{overundermicrosampling}, micro$-$sampling.  This experiment
uses the data from the first two experiments of Category III and attempts to create Which variants that use micro$-$sampling and compares them
to the Which2 that was created in Category III.

\sect{korudiagram} discusses a novel evaluation metric of software defect data that was created during the experimental process of this thesis.
\sect{which-exp} is not categorized in any of the categories in the above paragraph, but consists of a few experiments that were done with the
Which paremeters to justify our choices in the other experiments. \sect{t3} belongs to Category I.  \sect{multiclass} belongs to Category II.
\sect{defectmdp}, \sect{turkeydata}, and \sect{atntdata} all belong to Category III.  \sect{micro} is the final experiment and it belongs
to Category IV.

\section{Evaluation Metric: ``The Koru Diagram''}
\label{korudiagram}
\begin{figure}[!ht]
\begin{center}
\resizebox{100mm}{!}{\includegraphics{figures/chapter4/4.1/koru.png}}
\caption{Illustration of Main Components of Koru Diagram.}
\label{fig:koruexample}
\end{center}
\end{figure}
\fig{koruexample} is an example of the basic stucture of the Koru Diagram.  The Koru Diagram is a method of evaluating the performance
of different detectors generated by machine learners or other means, some of which are discussed below in \sect{defectlearners}\cite{korusupt2}.
The three lines in \fig{koruexample} correspond to the three detectors described in \sect{defectlearners}.  The way a detector is plotted
using this diagram is the key to understanding its evaulation.  The x$-$axis in the diagram is a normalized version of effort.  Effort
here can explained as the current total number of lines of code explored by this detector over the total number of lines of code
in the entire project.  This will be better described in \sect{defectdata}.  The y$-$axis is the probability of detection of the
detector being plotted. Koru et al. came up with the idea of these diagrams\cite{korusupt2}.  They stated that an effective measure of the 
performance of a detector would be to take all of the instances that the detector classifies as defective and sort them in ascending order based
on their line of code measure.  A plot of PD vs. Effort could then be created that shows the growth of the effort with relation to the grown of PD.

The curves of each detector on the Koru Diagram are a measure of how much effort is gained as defective modules are identified.  There
are a few things one can gain from this graph that are not explicitly defined in the plots.  For instance, if a curve in the Koru Diagram
has a lot of plateaus in which effort is growing but the probability of detection is not, one can see infer that the detector is reporting
non$-$faulty modules as faulty.  Another implicit observation that can be made will be explained in \sect{defectoracle}.  A third observation
that can easily be infered from the diagram and is illustrated in \sect{defect} is that most detectors do not find all defective  modules.
This is due to the fact that machine learners are not perfect and have to insist on some flaws to avoid over-fitting. The way this is reflected
in evaluation of the detector can be found in \sect{auc}.  A final observation of the Koru Diagram are the points 0,100 and 100,100.  No detector
will ever pass through or stop at the point 0,100.  This is because that point implies that all of the defective modules are found and no code has been
searched.  This is an impossibility because a module must have some source code associated with it.  The position 100,100 is special because any learner that
finishes at this point will have looked at every single module in the project.  The reason the manual and launam curves on \fig{koruexample} end there
is explained in \sect{defectmanup} and \sect{defectmandown}.

In order to generate the information to plot a curve on the Koru Diagram, one must take all of the modules that the detector classifies as
defective and sort them based on their lines of code in ascending order.  All curves start at point 0,0 on the graph.   From there
the effort and pd points are plotted such that for each defective module in the list, if that module is actually defective, increment the 
probability of detection by the fraction that one module is worth.  For every module that was classified as defective, increment the
effort by the portion of the effort that module adds to the current total.  So a point on the Koru Diagram can be interpreted as the amount
of effort this detector has incited to detect the number of modules it has detected.  A curve on the Koru Diagram is the path through the
space the detector takes to reach its goal.

The reason the modules classified as defective are sorted in ascending order is because faults are distributed disproportionately in smaller
modules \cite{quantitative}.  This obversation can be used as a method for an expert using the detector for their system to find a reasonable stopping point.
Essentially, if the distribution of the faults in a system follow the Pareto Distribtion described in \sect{pareto} as was observed in \cite{paretosrc1,paretosrc2,paretosrc3,korusupt1,analysis}
and repeated in the data sets that were used in \sect{defect}, than sorting the modules classified as defective in ascending order by the lines of code
should produce a curve most similiar to the oracle curve illustrated in \fig{koruexample}.  If the opposite is true, that is that the largest modules
contain the most faults and smaller modules have a far lower fault$-$density, than the oracle curve will have a shape similiar to the launam.
The curve will still stop at the same location in the diagram, but the shape will grow very slowly at first and then grow very quickly towards the final position.
This is the exact opposite of what is being illustrated in \fig{koruexample}.
\subsection{Special Detectors}
\label{defectlearners}
\subsubsection{Oracle}
\label{defectoracle}
The oracle curve described in this section can be seen illustrated in \fig{koruexample}.  The oracle is the perfect detector.  It is a purly hypothetic creation
and exists to illustrate the best possible detector that could be generated by a learner.  The oracle has one rule for detecting a defective module and that
rule is illustrated with \fig{oraclerule}.
\begin{figure}[!ht]
\ssp
\lstset{language=Python}
\lstset{backgroundcolor=\color{listinggray}}
\lstset{frameround=tttt}
\lstset{basicstyle=\scriptsize}
\lstset{showstringspaces=false, showspaces=false, showtabs=false}
\begin{center}
\begin{lstlisting}[frame=trBL]{}
if [ defect = true] then
	report defective module
endif
\end{lstlisting}
\end{center}
\caption{Oracle Detection Rule}
\label{fig:oraclerule}
\dbsp
\end{figure}
The main purpose of the oracle is to show the minimum possible effort required to detect all of the defective modules in a project.  As was observed in most of
the project data that was used in the experiments of \sect{defect}, the oracle grows very rapidly at first and then slows down after most of the defects have
been detected.  There will be no horizontal line in the oracle's curve because the detector that generated it has a probability of false alarm of zero.  The
reason for the slower growth towards the end is because the last remaining defects are located in very large modules with respect to the median of the module size.
Statistics taken about some of the projects that were used in \sect{defect} can be found in \sect{defectdatastats}.
\subsubsection{Manual$-$Up}
\label{defectmanup}
The manual$-$up detector in this section can be seen illustrated as the line titled manual in \fig{koruexample}.  A manual$-$up detector is a detector that does not
use a machine learner to generate a rule for each individual modules.  Instead, manual$-$up works by sorting all of the modules by lines of code in descending order
and exploring every one of the modules in that order. This, along with the manual$-$down detector described in \sect{defectmandown} are meant to be baselines of the
Koru Diagram.  If a detector cannot achieve a higher area than one of these two methods, it is not worth using.  An intersting thing to note about the manual$-$up
detector is that in most of projects used in the experiments of \sect{defect}, it loses out to manual$-$down.  This is due to the distribution of faults in the
modules that is described in \sect{pareto} and \sect{defectdatastats}. One article talks about the manual inspection method and its limits \cite{korusupt1}.  
This article discusses a type of inspection that is very similiar to our manual up inspection.
\subsubsection{Manua$-$Down}
\label{defectmandown}
The manual$-$down detector in this section can be seen illustrated as the line titled launam in \fig{koruexample}.  A manual$-$down detector works very similar to a
manual$-$up detector with one slight difference that makes a lot of difference in how well it performs.  The modules in the manual$-$down detector are sorted in
ascending order, like all of the other detectors in the experiments.  The causes the growth to of the curve to be more similar to the growth of the oracle.
\subsection{Area Under the Curve}
\label{auc}
The area under the curve for any curve on the Koru Diagram described in \sect{korudiagram} is a metric of evaulation for a given defect detector.  As was stated in
a previous section, not all detectors will end at the point of 100,100.  In fact, a detector ending here is an indicater of a poor detector. This is because
of the reasons described in \sect{korudiagram}.  The maximum area for any possible detector is the area of the oracle described in \sect{defectoracle}.  Upon
inspection of the Koru Diagram, it is easy to see that the oracle curve has very little area under it if it is taken to end at the point it actually does  end.
To ensure that the oracle does in fact have the  most area, the area under the line from the point the oracle ends, a point that will always be x,100, to
100,100 is added to the area of the oracle.  The same goes for all curves in this space.  If a curve ends at x,y, than the area under the line that begins at
x,y and ends at 100,y is added to the total area of the detector.  If a detector does not detect all defective modules in the project, than this will be reflected
in the area metric.  The reason to use a straight line is because this assumes the worst possible case of how a detector will work in the space it does not fire on.
That is it will miss all defective modules that exist in this space.  In order to effectively rank the areas under each detector's curve in this space, each area
is divided by the oracle.  This implies that the perfect detector will have a score of one. 
\section{Experiments with Which's Parameters}
\label{which-exp}
Which has some configurable parameters that it uses during its run that are mostly meant to change the run time of the application and
not the way the application reports and evaluates its rules.  These parameters can vary widely depending on the data Which is meant
to explore.  The following sections will describe different parameters and how Which uses these parameters to potentially change its run time.
These parameters were discusses under the implemention section of \sect{stoppingcond} earlier in \chap{which}.
\subsection{Changing the Maximium Selection Count}
\label{expmaxselectcount}
One of the main controls that determine the running time of Which and the potential quality of the rules that result from Which is the
maximum number of combinations that can be made before Which completes execution.  Which in its current state is an offline machine learner.
That is, it takes as input all of the data that it will be using to create a final rule.  Since a result is required in some reasonable time frame,
a simple solution is to only allow Which to combine so many times before stopping and reporting the rule.  This value should be set high enough
to allow a reasonable search of the search space.  \sect{expcheckparam} provides a solution with some experimental results in its favor that show
that there is no real reason to worry about finding a value high enough to reasonably search the space yet low enough to not require too much time.
\subsection{Changing the Check Parameters}
\label{expcheckparam}
Which employs a method of allowing to remove the burden of finding the optimal size for the maximum selection count described in 
\sect{expmaxselectcount} by allowing two parameters to be configured for every run of the application.  These parameters are known as
``check every'' and ``improvement.''  \sect{checkevery} and \sect{imp} below further describe what these two parameters do
and how they can  be used to gain a faster running time without sacrificing any of the quality of the rules.
\subsubsection{Check Every}
\label{checkevery}
Check every is a parameter of the application that allows the user to avoid a common ceiling effect that is shown in \fig{check}.
This effect can be attributed to what Menzies et. al. said in \cite{micro} about the ceiling effects of machine learners on static data.  This
parameter is used to control how often Which checks for a potential ceiling effect occuring in its list.  A ceiling effect can be defined as the
combonations that Which is creating to search the solution space either never having a better score or always having very similar scores.  This
results in Which never finding a better rule regardless of the number of combonations it makes.  This ceiling effect can occur very rapidly
and could result in a lot of the application's run time being wasted due to Which never finding a better rule than one just before the ceiling
happened.  A method of turning off this early stopping condition is to set the check every parameter to something that is greater than or equal to the
number of picks Which will do.
\subsubsection{Improvement}
\label{imp}
Improvement is a parameter of the application that allows the user to set how flat a ceiling has to be for it to be a ceiling.  In other words
improvement is what allows Which to determine if two rules are in fact the same in terms of their scores.  This number ranges from 0 to 1 and stands
for how much better the most recently generated rule is than one generated some time ago.  If this number is close to 0, Which is much stricter in
what it considers flat.  In fact, a value of 0 for improvement will disable this early stopping condition all together.  Improvement is tied to check every
in that the improvement of the best rule in the stack is queried every check every times a new rule is created via combination.

\begin{figure}[!ht]
\begin{center}
\begin{tabular}{cc}
	\resizebox{60mm}{!}{\includegraphics{figures/chapter4/ExperimentsWithWhichParams/CheckEvery/Plots/vowel.png}} &
	\resizebox{60mm}{!}{\includegraphics{figures/chapter4/ExperimentsWithWhichParams/CheckEvery/Plots/heart-c.png}} \\
	\resizebox{60mm}{!}{\includegraphics{figures/chapter4/ExperimentsWithWhichParams/CheckEvery/Plots/soybean.png}} &
	\resizebox{60mm}{!}{\includegraphics{figures/chapter4/ExperimentsWithWhichParams/CheckEvery/Plots/audiology.png}} \\
	\resizebox{60mm}{!}{\includegraphics{figures/chapter4/ExperimentsWithWhichParams/CheckEvery/Plots/autos.png}} &
	\resizebox{60mm}{!}{\includegraphics{figures/chapter4/ExperimentsWithWhichParams/CheckEvery/Plots/segment.png}} \\
\end{tabular}
\caption{Graphs Displaying the Early Maxima Phenomenon}
\label{fig:check}
\end{center}
\end{figure}

\subsection{Changing the List Size}
\label{listsize}
\subsubsection{Motivation for Experiments}
\label{listsizemotivation}
As stated in \sect{finitelist} in \chap{which}, the basic Which algorithm uses an infinite list.  What this means is that every rule, or detector,
that was ever generated exists in the sorted linked list.  This poses two problems.  The first problem is that if the number of picks is to be
very large, the number of rules generated will be very large.  This could cause the placement of each new rule to be a severe bottleneck in
some situations.  

The second problem is in the random probability of the selection algorithm.  The algorithm was meant to be so that a low scoring
rule would have less of a chance of being selected compared to a high scoring rule.  This uses the assumption that the combonation of high scoring
rules will lead to a desired optimal solution faster than combining any rules at random at every combonation.  This does, however, also still give
a chance that a low scoring rule will be selected in combonation.  This concept works under the assumption that a low scoring rule in conjunction
with another, probably high scoring rule, will cause a desirable result that could be overlooked.  The problem with having the list contain all
possible rules to select from is that the chances that any one of a number of low scoring rules being selected is higher after several combonations
have already been done.  This leads to Which possibly missing some better combonations because it becomes bogged down with selecting bad combonations
towards the end of the run.

A solution to both problems would be to limit the size of the sorted linked list that Which uses to keep track of all of the different combonations
generated so far.  The idea would be to only keep the top N best scoring rules.  This would make it so a worst case analysis of the Which algorithm
could be calculated.  Depending on the list size and number of overall combonations to be made, this could greatly enhance the run time of Which,
without losing the performance in terms of proper rule generation.  The last statement would only hold true if our assumption about combonations
of two high scoring rules will lead to better scores than combonations of low scoring rules with either high or low scoring rules.  If it is the
case that this does not hold, than we should see a significant performance decrease in the Which algorithm. \sect{listsizeexperiments} explains
the details of the experiments ran on this topic.
\subsubsection{Experiments}
\label{listsizeexperiments}
This experiment was done using six different sorted list sizes, descriptions of which can be seen in \tab{finitelistdesc}.  The method of training 
and testing on the data is explained in \sect{xval}. The type of evalation that was used for this experiment was the Koru Diagram described in
\sect{korudiagram}.  In this experiment, only Which was used as compared to itself in the native form of having an infinite list size.  This is because
this experiment was only meant to judge the performance, in terms of rule score, change that may or not occur due to changing the infinite list to
a finite one.  In \sect{listsizeres} below the results of this experiment are shown.

\begin{table}[!ht]
\begin{center}
\begin{tabular}{| r | l |} \hline
  \textbf{Learner} & \textbf{Description} \\ \hline \hline
  oracle & The perfect detector as described in \sect{defectoracle}. \\ \hline
  which2 & The standard infinite list version of Which. \\ \hline
  which10 & Which running with a maximum list size of 10. \\ \hline
  which20 & Which running with a maximum list size of 20. \\ \hline
  which50 & Which running with a maximum list size of 50. \\ \hline
  which100 & Which running with a maximum list size of 100. \\ \hline
  which1000 & Which running with a maximum list size of 1000. \\ \hline
\end{tabular}
\caption{Description of Different Learners Used in Experiment of \sect{listsize}.}
\label{tb:finitelistdesc}
\end{center}
\end{table}

\subsubsection{Results}
\label{listsizeres}
\fig{listsizegraphs} shows the results of some of the different finite list experiments.  These plots represent four different data sets
that the list was changed to be finite on.  The oracle is changing in these plots because the data being plotted is from different data sets.
These results show us that the infinite stack might actually not be necessary.  In most of these plots, it can be seen that the growth of Which
is almost identical to that of the growth of Which without the infinite stack.  The effects of forcing Which to have a finite stack on the run
time could be considerable in data sets where the attributes have a wide range of possible values.  In these types of data sets the time
it takes to find the position of new rule in the data set could be significatly altered if the stack were constrained to a small size independant
of the number of attribute values.

\begin{figure}[!ht]
\begin{center}
\begin{tabular}{cc}
	\resizebox{60mm}{!}{\includegraphics{figures/chapter4/ExperimentsWithWhichParams/FiniteList/ar4_3.png}} &
	\resizebox{60mm}{!}{\includegraphics{figures/chapter4/ExperimentsWithWhichParams/FiniteList/ar4_1.png}} \\
	\resizebox{60mm}{!}{\includegraphics{figures/chapter4/ExperimentsWithWhichParams/FiniteList/ar4_2.png}} &
	\resizebox{60mm}{!}{\includegraphics{figures/chapter4/ExperimentsWithWhichParams/FiniteList/ar5_2.png}} \\
\end{tabular}
\caption{Graphs Displaying the Effects of Changing the Sorted List Maximum Size. Which2 is always under Which1000 in these graphs.}
\label{fig:listsizegraphs}
\end{center}
\end{figure}

It cannot be seen in these graphs, but the standard infinite stack version of Which, Which2, always runs exactly the same as Which1000.  This
can be explained by the fact that Which uses the parameters of CheckEvery and Improvement to shorten its run time.  This causes the stack to
naturally never really exceed 1000.  While more experiments would need to be ran on the effects of making the stack finite, from these tests, 
it can be seen that the stack size being limited to a size of 10 to 100 actually improves the performance of Which in some cases and does not
really hurt the performance in the other cases.  We would recommend a stack size of 100 for most cases.  However, in situations where the number
of possible attribute values is very high, this number may too low.  It is important to consider the data that Which is being tuned to when
making these decisions.

The size of the stack contributes to the run$-$time of Which in two specific ways.  First, when attempting to place a newly created rule in the
stack it must compare the scores of each rule to the new rule until it finds the correct position.  In the worst case, over all $C$ rules
that are created, this process takes $\sum_{i=1}^C i = \frac{C(C+1)}{2}$ comparisons.  If the stack size is limited as it is shown here, this
number of comparisons is at worst the maximum size of the stack every time a new rule is created.  The second contribution to the run$-$time
is when two rules are being selected from the stack.  As was explained in \sect{selection}, Which will randomly select a rule based on its score.
If the stack size is very large, this process can hinder the run$-$time because every rule that is created requires two searches through the stack.
\tab{runtimetab} shows the differences in run$-$times for three different data sets of varying instance sizes.

\begin{table}[!ht]
\begin{center}
	\begin{tabular}{|c|c|c|c|} \hline
		\textbf{Stack Size} & \textbf{Data} & \textbf{Instances} & \textbf{Time$($seconds$)$} \\ \hline \hline
		$\infty$ & ar3 & 63 & 1.1 \\ \hline
		50 & ar3 & 63 & 0.9 \\ \hline
		1000 & ar3 & 63 & 0.6 \\ \hline
		$\infty$ & pc1 & 1109 & 9.2 \\ \hline
		50 & pc1 & 1109 & 9.8 \\ \hline
		1000 & pc1 & 1109 & 9.4 \\ \hline
		$\infty$ & mc2mod & 403 & 3.5 \\ \hline
		50 & mc2mod & 403 & 3.3 \\ \hline
		1000 & mc2mod & 403 & 3.8 \\ \hline
	\end{tabular}
\caption{Comparison of Run$-$Times for Different Stack Sizes.}
\label{tb:runtimetab}
\end{center}
\end{table}
\clearpage

\section{Category I: TAR3}
\label{t3}
As was discussed in \chap{which}, Which creates a rule that best decides one class in a data set.  This was shown to be the same type of
rule generation that TAR3 uses as well.  This experiment was designed to show if Which's simpler rule growing process is just as effective, or
more effective than TAR3's more complicated rule growing process.
\subsection{Data}
\label{t3data}
\begin{table}[!ht]
\begin{center}
\begin{tabular}{|r|c|c|c|}
	\hline
	\textbf{Name} & \textbf{\# Classes} & \textbf{``Best Class'' \%} & \textbf{\# Instances} \\ \hline \hline
	colic &  2 & 36.9565 &  726 \\ \hline
	kr-vs-kp &  2 & 47.7337 &  3351 \\ \hline
	breast$-$cancer &  2 & 29.4118 &  394 \\ \hline
	vowel &  11 & 9.09091 &  1201 \\ \hline
	anneal &  6 & 4.43951 &  1058 \\ \hline
	autos &  7 & 12.9808 &  336 \\ \hline
	audiology &  24 & 0.884956 &  298 \\ \hline
	vote &  2 & 38.3562 &  651 \\ \hline
	primary$-$tumor &  22 & 7.07965 &  596 \\ \hline	
	segment &  7 & 14.2857 &  2406 \\ \hline
	weather &  2 & 35.7143 &  23 \\ \hline
	weather.nominal &  2 & 35.7143 &  23 \\ \hline
	heart$-$h & 2 &63.95 & 294 \\ \hline
	heart$-$c & 2 & 54.4554 & 303 \\ \hline
\end{tabular}
\caption{Description of the UCI\cite{uci} data sets used in the experiment under \sect{t3}.}
\label{tb:ucidesctable}
\end{center}
\end{table}
\tab{ucidesctable} descibes the different UCI\cite{uci} data sets that were chosen for this experiment.  The data selected here was chosen for this
experiment because it covers a wide array of class counts and class distributions.  

\subsection{Experiments}
\label{t3design}
As was discussed in \sect{tar3}, TAR3 is a lift learner that will attempt to create a collection of rules that all attempt to predict the same
class.  For the purposes of this experiment, the ordering of the classes was chosen arbitrarily.  Essentially, the order the classes were listed
in the header of the file was the order of importance.

It was discussed in \chap{which} that the evaluation heuristic is completely independant of the rule growing processes in Which.  As such, we chose
to use the same evaluation heuristic for Which tha TAR3 uses, lift.  The equation for lift is seen in \eq{lifteqn}.

We decided that in order to compare the learners fairly, we would compare them using different discretization preprocessors.  We chose to use
equal frequency discretization with two bins, four bins, eight bins, sixteen bins, and thirty$-$two bins.  Part of our reasoning to go with the
two higher bin numbers was to see the possibility of Which naturally combining fine$-$grained bins into larger bins.  There were no conclusive
results in that area, however.

\cite{tar3} was the original thesis that created TAR3.  In that thesis, the author never discusses the performance of TAR3 under cross$-$validation.
As a result, part of this experiment was also to test how lift learning performs under cross$-$validation.  Cross$-$validation is discussed in
detail in \sect{xval}.  We decided to use the standard $10x10$ cross$-$validation for our experiment.

We report our results in the form of \textit{win$-$loss$-$tie tables} described in \sect{sod} from the perspective of Which.  That is, whenever
Which produces a rule that is better than TAR3's on the test data, Which receives a win.  In the opposite Which receives a loss.

\subsection{Results}
We report the results of the $12$ different data sets over the different discretization methods in \fig{tar3sod}.
\fig{tar3sodoverall} has the overall totals for each of the discretization policies.
\label{t3results}
\begin{figure}[!ht]
\begin{center}
	\input{tar3sod}
\caption{Sum of Differences Results for 2bins, 4bins, 8bins, 16bins, and 32bins comparison of Which and TAR3.}
\label{fig:tar3sod}
\end{center}
\end{figure}

\fig{tar3sod} shows the results of the experiments using the different bins in equal frequency discretizations. In these experiments it
appears that Which performed the best under a very coarse$-$grained discretization policy.  That is it performed better when less bins were
present for each attribute.  This could be caused by the overall search space being smaller.  A data file with attributes ranging in only two or
four values takes far less time to search exaustively than one with attributes ranging with thriry$-$two values.  This means that the random
nature of Which has a greater chance to not find an optimum solution.

It is important to note that Which and TAR3 tied far more than Which won and still more than when they tied.  This means that the simpler
approach to rule construction that Which has is still effective in terms of lift learning.  This goes with the idea that simpler learners perform
just as well as more complicated ones that Dr. Menzies et al. say in \cite{staticlearn}.

\fig{tar3sodoverall} provides the total results from the five differenent discretization policies in this experiment.  While Which has nearly
two times as many wins at it does losses, the majority of time Which tied with TAR3. In $\frac{207}{679} \approx 30.5\%$ of the tests ran, Which
created a better rule that TAR3, in $\frac{80}{679} \approx 11.8\%$ of the test ran, Which created a poorer rule than TAR3, and in
$\frac{392}{679} \approx 57.7\%$ of the test ran, Which and TAR3 created a rule with the same score. These overall results do show that while Which does
not create better rules over TAR3 all the time, $88.2\%$ of the rules it creates are the same or better than TAR3.  This agrees with \cite{staticlearn}
perfectly in that simpler learners can and do perform very well when compared to more complicated ones.

\begin{figure}[!ht]
\begin{center}
	\input{tar3sodoverall}
\caption{Overall Sum of Differences for Which vs TAR3.}
\label{fig:tar3sodoverall}
\end{center}
\end{figure}
\clearpage

\section{Category II: Multi Class Classification}
\label{multiclass}
The purpose of this experiment was to show how Which performed in a situation where more than two classes were present in the data sets.
The data sets used in this experment come from the UCI\cite{uci} data sets listed in \tab{ucidesctable}.  The data sets that were selected were selected
solely on the criteria that they had more than two classes.  This was because a data set that has only two classes can produce a high accuracy
by interpreting the rule that Which generates as having an added else clause.  For instance, Which may generate a rule like \fig{multiclasswhichrule}.
In the case of only having two classes, this could be understood as, ``if that rule, then class X, else class Y.''  However, in the case of having
more than two classes, the rule could only be interpreted as, ``if that rule, then class X, else no answer.''  This leads to extreme problems in the
prediction quality of the rule.  As described in \chap{which}, Which produces rules that best predict the ``best class'' in a data set.  Best class
is meaningly when the goal of test is to create something that will predict the class that new instances in a data set fall into.  However, in the case
of most of these UCI data sets, there is no best class, that is the classes are all equally valued.  So selecting one of these classes as the best
is completely artificial.

\begin{figure}[!ht]
\ssp
\lstset{language=Python}
\lstset{backgroundcolor=\color{listinggray}}
\lstset{frameround=tttt}
\lstset{basicstyle=\scriptsize}
\lstset{showstringspaces=false, showspaces=false, showtabs=false}
\begin{center}
\begin{lstlisting}[frame=trBL]{}
if [ outlook = overcast ]
AND temp = [ high OR low ]
\end{lstlisting}
\end{center}
\caption{Oracle Detection Rule}
\label{fig:multiclasswhichrule}
\dbsp
\end{figure}

This experiment was created to show an area that the current version of Which fails in.  It was meant to illustrate a counter to the experiments of
\sect{defectmdp} and \sect{turkeydata} discussed in this chapter.  Essentially, Which fits into a niche portion of the data mining world.  A
possible solution for this problem in Which is discussed in \chap{futurework}.

\subsection{Which's Heuristic}
\label{muilticlasswhichh}
A common metric used for evaluating a decision making process that is created by a machine learner is \textit{accuracy}.  The equation for accuracy
can be seen in \eq{accuracy}.  In this equation, $TP_{i}$ stands for the number of true positives of the $i^{th}$ class.  This is the number of the
$i^{th}$ class that was accuratly classified as the $i^{th}$ class.  The n in the summation is the number of class.  Finally, the $class_{i}$ in the
denomenator of the equation is the sum of all the classes in this data set.  Accuracy effectively depicts the total number of classes that were
classified correctly.  
\begin{center}
\begin{equation}
	\displaystyle h_{accuracy} = \frac{\sum_{i=1}^n TP_{i}}{\sum_{i=1}^n class_{i}}
\label{eq:accuracy}
\end{equation}
\end{center}
There are some problems associated with accuracy that are described in \cite{blindspot}.  For example, if there is
a data file that has one class that is very infrequent in the data set and a learner never correctly classifies that class, the accuracy may still
be very high.  The following is an example of a two class data file that contains 100 instances.  One of the classes is 99 of the instances whereas
the other is only one instance in the set.  In this two class example, one class consists of 99\% of the total distribution.
If a machine learner produces the rule that classifies everything as class X, it will have an accuracy of 99\%.  The major problem of this is if
the class that represents 1\% of the population is the class we are trying to predict for, that information is entirely lost even though the
detector is 99\% accurate.  This type of detection, isolating one class and developing a rule for that class, is called lift learning and described in
\sect{t3}.  This is also the type of learning that Which uses in its algorithm, as said in \chap{which}, only one rule is produced and that is the
rule that best predicts a ``best class.''  Which should and does fail in this accuracy over a multiple class data set test.
\subsection{Design of Experiments}
\label{multiclassdesign}
In order to do this experiment, only a few of the data sets from \tab{ucidesctable} were chosen.  This data sets all had one thing in common and that
was that there were more than two classes in the data set.

Once the data was chosen a discretization method known as FayyadIrani\cite{fayyadirani},
discussed in \chap{related}, that was implemented in the wEka\cite{weka} was used as a preprocessor to allow each of the learners to be on the same playing field.  What is meant by this
is that each of the learners have their on discretization methods in them.  Using FayyadIrani as a preprocessor was a way of ensuring that each of the learners
were actually learning from the same data.  

After discretization was complete, the data was broken in into three train and three test sets with a process that is described in \sect{xval}.  The learners
where ran using train n and test n and the results were complied into MannWhitney U$-$Tests.  There was one test generated for each of the individual
files and an overall test on all of the scores.
\subsection{Results}
\label{multiclassresults}
\tab{mwumulticlass} contains the overall MannWhitney U test results for these learners on the data sets.  It can be seen from these results that
j48, \nbc, and ripper all perfrom the same on all of the data,  they all tie with each other.  It can be shown also that Which is completely outperformed
in this environment as well.  
\begin{table}
\begin{center}
\begin{tabular}{|r|c|c|c|c|}
        \hline
        \textbf{key} & \textbf{ties} & \textbf{win} & \textbf{loss} & \textbf{win-loss-at-99\%} \\ \hline \hline
        nBayes & 2 & 1 & 0 & 1 \\ \hline
        jRip & 2 & 1 & 0 & 1 \\ \hline
        j48 & 2 & 1 & 0 & 1 \\ \hline
        which & 0 & 0 & 3 & -3 \\ \hline
\end{tabular}
\caption{Overal MannWhitney U$-$Test on Selected UCI Data Sets.}
\label{tb:mwumulticlass}
\end{center}
\end{table}
\fig{quartilemulticlass} illustrates further the results of this experiment.  Quartile charts are discussed in detail and used to report results
in \cite{staticlearn}.  Quartile charts have certain properties about them that make them valuable for reporting results of cross validation
experiments.  First of all, so called standout results are clearly visable in these charts.  Looking at the results in the MannWhitney U test shows
that Which did lose to all three other learners.  However, the magnitude of how poorly it performed compared to the other learners is not captured
in this table.  \fig{quartilemulticlass} illustrates this very well in that the maximum accuracy score for Which over all the tests is 57.1.  Another
visual cue to the performance using the quartile chart is the median value, the black circle in the figure.  It can be seen clearly in this
chart that Which's median is zero, since no circle exists on the chart.  The other thing about the medians that can be gained from looking at this
is that the other three learners have similar medians, and also that all of these medians are higher than the maximum performance of Which.  Another
propertiy of the quartile chart is that it makes no parametric assumption about the underlying distribution of the data like t$-$tests\cite{ttest}.  Further
explaination of the u$-$test, t$-$test, and quartile charts can be found in \chap{related}.

\ssp
\begin{figure}[!ht]
\begin{center}
\begin{tabular}{|ccc|c|}
	\hline
	\textbf{Learner} & \textbf{Mean} & \textbf{Equal} & \textbf{Description} \\
	\hline
	nBayes & 82.9 & & \boxplot{40.7}{26}{82.9}{91.3}{7.3} \\
	j48 & 79.7 & $=$ & \boxplot{33.6}{40.7}{79.7}{93.0}{6.4} \\
	jRip & 79.7 & $=$ & \boxplot{31.9}{35.2}{79.7}{93.4}{5.7} \\
	which & 0.0 & $<$ & \boxplot{0.0}{0.0}{0.0}{13.1}{44.0} \\
	\hline
\end{tabular}
\caption{Quartile Chart Illustrating the Results of \tab{mwumulticlass}.}
\label{fig:quartilemulticlass}
\end{center}
\end{figure}
\dbsp

These results were expected before the experiment and, in actuality, the experiment was designed to illustrate this point of the Which design.
The reason Which does so poorly with a multi class data file is because by design, it is unable to generate rules for each of the classes.  It can
only generate rules for one class.  While it may be possible to create a tool around Which that runs the Which application once for each class in the
data file, that is beyond the scope of this thesis.  Further improvements to Which can be found in \sect{futurework}.
\clearpage

\section{Defect Detection}
\label{defect}
One of the most impressive applications of the Which learner is that of defect detection in large software projects.
Previous studies have shown that \nbc, described in \sect{bayes}. has been superior at
detection compared to the other classical learners \cite{staticlearn}.  As was stated previously, there were three different experiments on the
performance of the machine learners using defect detection data.  These experiments all used different data  Each collection of data is called
a family in these experiments.  The MDP family contains all of the data sets used in \sect{defectmdp}, the Turkey family contains all of the data
used in \sect{turkeydata}, and the \att family contains all of the data used in \sect{atntdata}.
\subsection{Data}
\label{defectdata}
\subsubsection{Basic Statistics Taken About Data}
\label{defectdatastats}
\begin{table}[!ht]
\begin{center}
\begin{tabular}{| r || l | l | l | l | l | l |} \hline
	\textbf{Data} & cm1  & kc1 & pc1 & ar3 & ar5 \\	\hline
	\textbf{Family} & mdp & mdp & mdp & Turkey & Turkey \\ \hline \hline
	\textbf{Modules} & 498 & 2109 & 1109 & 63 & 36 \\ \hline
	\textbf{Median} & 17 & 9 & 13 & 42.5 & 42 \\ \hline
	\textbf{Mean} & 29.6448 & 20.3723 &  23.3761 & 89.2698 & 75.889 \\ \hline
	\textbf{$\sigma$} & 42.7106 & 29.7474 & 35.2681 & 115.802 & 88.8691 \\ \hline
	\textbf{Min} & 1 & 1 & 0 & 3 & 5 \\ \hline
	\textbf{Max} & 423 & 288 & 602 & 670 & 477 \\ \hline
	\textbf{Defect Dist} & 9.8\% & 15.5\% & 6.9\% & 12.7\% & 22.2\% \\ \hline
\end{tabular}
\caption{Statistics Taken About LOC From a Few of the Data Sets Used.}
\label{tb:datastattable}
\end{center}
\end{table}

The data represented in \tab{datastattable} was taken from five of the twelve data sets that were used in the defect detection tests.  The data sets used
to create two of the three of the defect detection experiments were taken from the PROMISE website \cite{promise}.  Cm1, kc1, and pc1 are part
of the mdp family and come from NASA software projects.  The Turkey data comes from a software company that creates control software for
household appliances.  The third experimental data comes from \att. This data was used under a non disclosure agreement and not much is allowed
to be said about it.  The results of the experiment and a discription of some of the attributes used.  This table is meant to give a brief overview
of some of the statistics on the the project files that were used.  Refer to \sect{defectmdp} and \sect{turkeydata} for more information about the files
and types of projects these data files were created from.
\subsection{Halstead and McCabe}
\label{halsteadmccabe}
\begin{table}[!ht]
\begin{center}
\begin{tabular}{|l | l | l|} \hline
	\textbf{Metric} & \textbf{Metric} & \textbf{Definition} \\
	\textbf{Type} & & \\ \hline \hline
	McCabe & v(G) & Cyclomatic Complexity \\
		   & ev(G) & Essential Complexity \\
		   & iv(G) & Design Complexity \\
		   & LOC & Lines of Code \\ \hline
	Derived & N & Length \\
	Halstead & V & Volume \\
			 & L & Level \\
			 & D & Difficulty \\
			 & I & Intelligent Count \\
			 & E & Effort \\
			 & B & Effort Estimate \\
			 & T & Programming Time \\ \hline
	Line & LOCode & Lines of Code \\
	Count & LOComment & Lines of Comment \\
		  & LOBlank & Lines of Blank \\
		  & LOCCodeAnd & Lines of Code \\
		  & Comment & and Comment \\ \hline
	Basic & UniqOp & Unique Operators \\
	Halstead & UniqOpnd & Unique Operands \\
			 & TotalOp & Total Operators \\
			 & TotalOpnd & Total Operands \\ \hline
	Branch & BranchCount & Total Branch Count \\ \hline
\end{tabular}
\caption{Halstead and McCabe Descriptions of MDP Data Sets.}
\label{tb:mdpdatadesctable}
\end{center}
\end{table}
\cite{randomforest} contains a table that describes all of the different attributes used in both the MDP and the Turkey data sets.  
\tab{mdpdatadesctable} is the table that was used in that article. All of these statistics are taken on a per module basis. In this instance,
a module is the simplest whole unit in the language.  In C this would be a function and in Java this could be considered a method 
of a class.  This is not to be confused with the broader term of module that is discussed in \cite{ostrand1}.  In that article, it is said 
that the file is far more fine grained than a module.  Their definition of module is something similiar to a collection of Java classes that 
are a stronly cohesive, strongly coupled system.  Here, we use the term module to be something that is more fine grained than a file.

There are three types of statistics that are taken about each module, these are the McCabe \cite{mccabe, testless,randomforest}, Halstead \cite{halstead,testless,randomforest}, and line of 
code readings.  

McCabe's idea was that it was the complexity in the pathways of the source code that would lead to more faults.  The three
attributes that are associated with his idea can be seen in \tab{mdpdatadesctable}. Cyclomatic complexity can be described as the number of
linearly independant paths that can be taken through source code.  A simple explanation of this would be a simple module that contained one if 
statement, or conditional.  This would have a cyclomatic complexity measure of 2.  Essential complexity refers to the cyclomatic complexity of
a module after all well structured paths have been reduced to statements.  For instance, unrolling a for or while loop into its statements. Design
complexity of a module is the cyclomatic complexity of its simplified form.

Halstead's idea was that if the code was hard to read, it would have more faults.  Halstead took measures about the code itself, instead of the
flow of the program.  These attributes can be seen in \tab{mdpdatadesctable}.

The final measurements taken about the modules are basic line of code readings.  These give specific line count readings about the code and its
executable.

\subsection{Design of Experiments}
\label{defectdesign}
\subsubsection{Logging the Attributes}
\label{lognums}
A plot of the attributes in a few of the data files reveals that they fall in an exponential distribution.  This has been stated before in
\cite{staticlearn}.  Three of these plots can be seen in \fig{expplot}.  This type of distribution is difficult to learn from, especially when
using an equal frequency discretization policy like Which uses in these experiments.  What an exponential distribution causes is that, in a 10 bins
example, most of the bins actually contain the same range of values, since these values are the most plentiful.  It is only the last one or two
bins that contain any meaningful information.  Because of this, the machine learner will always pick the last bin.  This type of selection
leads to several false alarms.  To prevent this behavior, we took the natural log of each of the attributes in the data file.  This gives a linear
slope and allows for the equal frequency binning to better represent the values that are contained in the bins.  As was discussed in
\sect{equalfrequencysect}, the result of equal frequency discretization on an exponentially distributed attribute is that the majority of the
lower bins contained values in the range $[0,1)$ and several of the initial bins contained the same range of values.  Which and the classic learners
would always create detectors that selected the higher bins, which typically contained values that had a range that was $\geq 100$.  Taking the
natural log of these attributes provided much more similiar ranges.  The equation that was used to handle the transformation of the attributes was
$newval = ln(max(oldval, 0.0001))$.  This was to avoid the cases where the attribute value was zero.
\begin{figure}[!ht]
\begin{center}
\begin{tabular}{cc}
	\resizebox{60mm}{!}{\includegraphics{figures/chapter4/lognums/plots/kc3_mod.png}} &
	\resizebox{60mm}{!}{\includegraphics{figures/chapter4/lognums/plots/mw1_mod.png}} \\
	\resizebox{60mm}{!}{\includegraphics{figures/chapter4/lognums/plots/mc2_mod.png}}
\end{tabular}
\caption{Some Example Attribute Distributions for kc3mod, mw1mod, and mc2mod.  Here, we sorted the values of the attributes in each data set in
ascending order.  What we are measuring is the distribution of the values over the number of instances in the set.}
\label{fig:expplot}
\end{center}
\end{figure}

\subsection{Which's Heuristic}
\label{defectwhichh}
As discussed in \sect{scoring} earlier in this writing, Which allows any scoring heuristic to determine how a generated rule scores.  Several
different scoring heuristics were tried in the tests, but the one discussed in this section is the heuristic that was used for the defect
detection experiments.
\subsubsection{Balance}
\label{balance}
\eq{balanceeqn} illustrates the heuristic that Which uses to calculate the score of each rule.  For our purposes, the best rule would be one
that has a 100\% probability of detection and a 0\% probability of false alarm, this corresponds to the oracle discussed in \sect{defectoracle}.
Because of this, the balance equation is the Euclidean distance from that point in ROC space.  Scalar terms of $\alpha$ and $\beta$ are added 
to the equation to allow for different levels of importance for each of the terms.  By default, these results are 1 and 1000 respectively.  What
this does is make a very low pf 3 orders of magnitude more important than the pd term.  Having $\alpha=1$ and $\beta=1000$ was decided upon after
considering the effects of both pd and pf.  We wanted to find a way to include effort into the heuristic, but effort in itself cannot be included
explictly in the heuristic.  This is because or reasons discussed further in this section and also because pd and effort are correlated in a very
distinct way.  In order to detect a module, some effort must be used.  Having the $\alpha$ and $\beta$ coefficients be set to what they are for
experiment is a way of implicitly saying that while effort must be tolerated, minimizing the contribution to effort from false alarms should
be a priority.
\begin{center}
\begin{equation}
h_{balance} = \frac{\sqrt{pd^2*\alpha + (1-pf)^2*\beta}}{\alpha+\beta}
\label{eq:balanceeqn}
\end{equation}
\end{center}
As was stated earlier in this section, an equal contributor to how well a detector scores is how much effort is required to detect a certain 
percent of the defects in a project.  This metric was added to the balance equation to make it look like \eq{balancewithefforteqn} and the 
results for Which were not ideal.  As before, there are scalar terms added to the equation to allow for importance of each part to be specified.
\begin{center}
\begin{equation}
h_{balancewitheffort} = \frac{\sqrt{pd^2*\alpha + (1-pf)^2*\beta + (1-effort)^2*\gamma}}{\alpha+\beta+\gamma}
\label{eq:balancewithefforteqn}
\end{equation}
\end{center}
The typical rule that was selected using the heuristic of \eq{balancewithefforteqn} was a middle ground.  What this means is that the best
rule that Which could report had a pd of 0.5, a pf of 0.5 and an effort 0.5.  This is because the way in which effort and pd are correlated.
While it is possible for a rule to be at the 0,1 mark in ROC space, it is impossible for a rule to be at the 0,1 mark in Koru space.  This is
because detecting a module as defective will always add some degree of effort to the rule.  This causes the phenomenon of finding the middle
ground.  Also, even though effort is not evaluated by the rules during the growing phase, pf is given a really high priority, thus inducing
a minimal effort.  As described in \sect{korudiagram}, if a detector has a high pf than the overall effort will be very high whereas if a 
detector has a very low pf it will more closely follow the oracle curve.
\subsubsection{Probability of Detection Divided by Lines of Code}
\label{pdoverloc}
\eq{pdoverloceqn} illustrates how a rule is scored using this heuristic.  It was decided
before these experiments took place that since it was easy enough to change the scoring
method of Which, that we would also try a different scoring heuristic.
\begin{center}
\begin{equation}
h_{\frac{PD}{LOC}} = \frac{PD}{\sum_{i=1}^n LOC_{i}}
\label{eq:pdoverloceqn}
\end{equation}
\end{center}
\eq{pdoverloceqn} was meant to create rules that have a high probability of detection
and a low effort total.  Instead of actually using effort here, which is a normalized amount
of the lines of code totally looked at over the total lines of code, this equation
uses the raw line of code measures.  In \eq{pdoverloceqn} PD stands for the probability
of detection and the summation in denomenator stands for the number of modules that were
labeled as defective by a rule.  $n$ is the total number of modules labeled as defective.

$h_{\frac{PD}{LOC}}$ was used because it was slightly different than $h_{balancewitheffort}$
and we had hoped it would yield the kind of results that we wanted.  That is, results that
have a high probality of detection with a low amount effort required.  Another property of this
equation is that it could allow for modules to be labeled as erroneous that were not actually
erroneous, but if that module was too large in terms of lines of code, the score would be
significantly impacted compared to a smaller module. The results of this heuristic are discussed
in \sect{defectresults}.
\subsection{Evaluation Criteria}
\label{defecteval}
The evaluation criteria for all of the detectors used in these experiments is the evaluation criteria described in \sect{auc}.  Each detector
is plotted  in Koru space and then its normalized area under the curve is used for the Mann Whitney U$-$test.  With the exception of the \att
data used in \sect{atntdata}, the other experiments all used a cross validation.  The reasons the \att data did not are discussed in section discussing
the \att experiment.
\subsection{Probability of Detection vs. \%Lines of Code}
\label{defectpdvseffort}
Probability of Detection vs. \%Lines of Code is a general name for the Koru diagram.  With
each of the experiments in \sect{defectmdp}, \sect{turkeydata}, \sect{atntdata}, and \sect{micro}, 
this is both the method of displaying the results as well as the method of scoring the
results. Several of the figures in this thesis use this type of graph.
\clearpage

\section{Category III: MDP Data}
\label{defectmdp}
The data taken from the MDP\cite{promise} refers to data used in various NASA programs.  This infers that this data was taken from a company
that is not profit driven and creates software via contractual agreements.  This means that the different projects were actually developed by
different companies from geological regions spanning the United States.  Some of these projects were created from entirely new source code
while other projects were created by modifying and reusing another project's code.  The individual project files for this data each represent
a different project and potentially a different team of developers.  While the source code underlying the statistics gathered for these data files
is still structured relatively the same use the standard government contract standards of producing code, the projects themselves are still diverse
enough to make the claim that they are not the same in terms of programming style.  This means that performing well on these project files overall
creates some support for the fact that a detector or machine learner generating a detector could have some stability over projects that are
diverse and not of the same type.  \tab{nasainfo} describes some statistics about the seven data sets, or project files, that were used
in the the MDP experiments.
\begin{table}[!ht]
\begin{center}
	\begin{tabular}{|r|c|c|c|}
		\hline
		\textbf{File Name} & \textbf{Language} & \textbf{Defects} & \textbf{Modules} \\ \hline \hline
		cm1 & C$++$ & 49 & 498 \\ \hline
		kc1 & C$++$ & 326 & 2109 \\ \hline
		kc2 & C$++$ & 107 & 522 \\ \hline
		mw1\_mod & C$++$ & 31 & 403 \\ \hline
		pc1 & C$++$ & 77 & 1109 \\ \hline
		kc3\_mod & JAVA & 43 & 458 \\ \hline
		mc2\_mod & C$++$ & 52 & 161 \\ \hline
		\hline
	\end{tabular}
\caption{Information About the NASA data.}
\label{tb:nasainfo}
\end{center}
\end{table}

\subsection{Experiment}
\label{defectexp}
This experiment was done on seven different MDP data sets using a $10x3$ cross$-$validation on
nine different learners plus the oracle, manualUp, and manualDown detectors.  The total number
of times this each data file was used is $7x10x3x12=2520$.  The statistics taken on each
of the detectors generated were the probability of detection and the total effort used after
all of the modules the detector said were defective were inspected.  In order to get
the specific data for the Koru Diagram and for the Area Under the Curve calculations, a
step by step of each detected defective module was recorded as well.  This process was
explained in \sect{korudiagram}.

Several variants of Which were used in these experments and some appear in the graphs while
all appear in the MannWhitney and Quartile Chart reports.  They were omitted from the graphs
because they all performed poorly and added unnecessary clutter and confusion to them. A
discription of these different variants can be seen in \tab{whichvartab}.  In all of these
experiments, Which2 was the majority best performing variant, so only it is plotted in the
graphs of \fig{mdpreswhichmanup}, \fig{mdpreswhichall},  and \fig{mdprescat2qtc}.

\begin{table}[!ht]
\begin{center}
\begin{tabular}{|r|l|}
	\hline
	\textbf{Variant Name} & \textbf{Description} \\ \hline \hline
	Which2 & Which using 2 bins equal frequency with \eq{balanceeqn}.\\ \hline
	Which4 & Which using 4 bins equal frequency with \eq{balanceeqn}.\\ \hline 
	Which8 & Which using 8 bins equal frequency with \eq{balanceeqn}.\\ \hline
	Which2loc & Which using 2 bins equal frequency with \eq{pdoverloceqn}.\\ \hline
	Which4loc & Which using 4 bins equal frequency with \eq{pdoverloceqn}.\\ \hline 
	Which8loc & Which using 8 bins equal frequency with \eq{pdoverloceqn}.\\ \hline
\end{tabular}
\caption{Descriptions of Which Variants used in \sect{defectexp}.}
\label{tb:whichvartab}
\end{center}
\end{table}
\subsection{Results}
\label{defectresults}
We have divided these results into three seperate classifications to illustrate different
points.  The three classifications are:
\begin{itemize}
	\item Class One: Which2 $>$ manualUp $>$ classic learners;
	
	\item Class Two: Which2 $>$ everything else; and,
	
	\item Class Three: Everything $>$ Which
\end{itemize}
Each class consists of three figures that correspond to each other.  The first figure
is a Koru Diagram, explained in \sect{korudiagram} for each of the data sets that are in
the category.  The second figure represents the quartile charts for the graphs.  The charts
are in the same layout as the Koru Diagram.  The third and final figure is the
figure representing the MannWhitney U$-$Test tables that are generated from the same
data the quartile charts are generated from.  These, like the quartile charts, are
in the same layout as the Koru Diagram.  By relating the three figures to each other, it
is possible to see each of the different categories.

Sometimes, the quartile chart will report something different than the MannWhitney U$-$Test
table.  What we mean by this is that the two charts report their learners in descending
order based on how they score.  However, it can be seen by inspecting both the MannWhitney
and the Quartile Chart together to determine the category each of the results is in.
\subsubsection{Class One}
\label{defectrescat1}
The first class is the one where Which2 has the best overall score, followed
by manualUp, followed by the rest of the learners.  This class can be seen graphically
in \fig{mdpreswhichmanup}.  We feel this class is significant because it illustrates
two important points.  The first point is that Which2 was the overall winner and it
can be seen in \fig{mdpreswhichmanup}.  The second point is that the manual detector
manualUp, described in \sect{defectmanup}, performs better than any of the classic learners.
This says that the detectors generated by these learners gives us nothing better than
what manually checking all of the source code in this manual up way, which is sorting
the modules in ascending order by lines of code only.

\fig{mdprescat1qtc} and \fig{mdprescat1mwu} display the graphs in \fig{mdpreswhichmanup}.
\fig{mdprescat1qtc} represents the Quartile Charts that were used in categorizing the the
the different effects that were realized.  Looking at the quartile charts shows three important
facts.  The first fact is that very few of the machine learners actually performed well.  Of these
learners, Which2 had the best performance, followed by manualUp. which is the definition of the category.
However, the second point is that \nbc performed better than the other standard machine learners.  This
results is consistend with the results in \cite{staticlearn}.  The third point of interest in these results is
that manualDown also performs better than all of the standard learners besides \nbc.
\begin{figure}[!ht]
\begin{center}
\begin{tabular}{cc}
	\resizebox{60mm}{!}{\includegraphics{figures/chapter4/MDP/plots/cm1.png}} &
	\resizebox{60mm}{!}{\includegraphics{figures/chapter4/MDP/plots/kc1.png}} \\
	cm1 & kc1 \\
	\resizebox{60mm}{!}{\includegraphics{figures/chapter4/MDP/plots/kc2.png}} &
	\resizebox{60mm}{!}{\includegraphics{figures/chapter4/MDP/plots/mw1_mod.png}} \\
	kc2 & mw1mod \\
	\resizebox{60mm}{!}{\includegraphics{figures/chapter4/MDP/plots/pc1.png}} & \\
	pc1 & \\
\end{tabular}
\caption{MDP Data Files where Which2 $>$ manualUp $>$ rest.}
\label{fig:mdpreswhichmanup}
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
	\input{qtccat1}
\caption{Quartile Charts Repesenting Graphs in \fig{mdpreswhichmanup}}
\label{fig:mdprescat1qtc}
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
	\input{mwucat1}
\caption{MannWhitney U$-$Tests Representing Graphs in \fig{mdpreswhichmanup}}
\label{fig:mdprescat1mwu}
\end{center}
\end{figure}

\subsubsection{Class Two}
\label{defectrescat2}
The second class the one in which Which2 $>$ everything else, but manualUp is beaten
by some of the classic learners.  This classification illustrates two points that we found
important.  The first one is that Which2 wins overall, another victory for Which2, and
that some of the detectors generated in this class perform better than the manual inspection.
This second point is illustrating that manualUp should not be the current de facto
standard. 

A point of interest here is that \nbc performs better than the other standard learners in this class, much
like class one.  This is still consistent with the results of \cite{staticlearn}.
\begin{figure}[!ht]
\begin{center}
\resizebox{60mm}{!}{\includegraphics{figures/chapter4/MDP/plots/kc3_mod.png}}
\caption{MDP Data Files where Which2 $>$ rest. This is for kc3 mod.}
\label{fig:mdpreswhichall}
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
	\input{qtccat2}
\caption{Quartile Charts Representing Graphs in \fig{mdpreswhichall}.}
\label{fig:mdprescat2qtc}
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
	\input{mwucat2}
\caption{MannWhitney U$-$Tests Representing Graphs in \fig{mdpreswhichall}.}
\label{fig:mdprescat2mwu}
\end{center}
\end{figure}

\subsubsection{Class Three}
\label{defectrescat3}
The third and final class is the one in which manualUp wins overall, and the classic
learners perform better than Which2.  This class contains only one file and it
also represents the one time out of the eight that Which2 does not perform well.
\begin{figure}[!ht]
\begin{center}
\resizebox{60mm}{!}{\includegraphics{figures/chapter4/MDP/plots/mc2_mod.png}}
\caption{MDP Data Files where all $>$ Which.  This is the data file, mc2\_mod.}
\label{fig:mdpreswhichlose}
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
	\input{qtccat3}
\caption{Quartile Chart Representing Graphs in \fig{mdpreswhichlose}.}
\label{fig:mdprescat3qtc}
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
	\input{mwucat3}
\caption{MannWhitney U$-$Tests Representing Graphs in \fig{mdpreswhichlose}.}
\label{fig:mdprescat3mwu}
\end{center}
\end{figure}

\subsubsection{Overall}
\label{defectresoverall}
\fig{mdpoverallqtc} and \fig{mdpoverallmwu} show the overall results of the experiment.  These tables of data contain information
about the experiment that give an overview over all of the NASA projects used in the MDP experiment.  There are a few things that can be gathered
from these figures.  The first is that Which2 wins uncontested over all of the projects according to the MannWhitney U$-$Tests.  This means that,
overall, Which2 has the best performance over all of the other detectors.  Another point of interest is that \nbc performs better than the
other classic machine learners overall.  This is the same assertion that was made in \cite{staticlearn}.

It can be seen that our second heuristic for Which, $h_{\frac{PD}{LOC}}$, did poor no matter the discretization policy.  However, although
Which4 and Which8 performed much poorer than Which2, they both still performed better than j48 and Ripper.  The fact that Which2 performed
the best overall and the best out of all of the Which variants says something about that data.  As stated before, Which2 uses a two bins, equal
frequency discretization policy.  This means that all of the attribute values are sorted in ascending order, and the first 50\% are in bin one, while
the second 50\% are in bin two.  This kind of discretization effectively does a binary dissection of the data.  The fact that Which performs best
using this kind of discretization implies that the attributes themselves, after taking the natural log of the attributes, tend to predict the class
based on their value.  At least the attribute values that Which2 chose in its detection policy tend to predict if a module is defective or not in
a very simple way.  It could be said that after or before some threshold value of a few of the attributes, a module tends to be defective.

\begin{figure}[!ht]
\begin{center}
	\input{mdpoverallqtc}
\caption{Quartile Charts Representing the Overall Performance of the Detectors on the MDP Experiment.}
\label{fig:mdpoverallqtc}
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
	\input{mdpoverallmwu}
\caption{MannWhitney U$-$Tests Representing the Overall Performance of the Detectors on the MDP Experiment.}
\label{fig:mdpoverallmwu}
\end{center}
\end{figure}
\clearpage

\section{Category III: Turkey Data}
\label{turkeydata}
In order for us test the external validity of the results of \sect{defectmdp}, we
chose a completely different type of software data.  The data from a Turkish appliance company and was chosen just for this purpose.
The Turkey data is in three seperate files that are each for a different type of appliance.  \tab{turkeyinfo} describes some
properties about the data.  An important point to gain from this table is that the number of actual modules in these projects
is significantly smaller than the projects in \tab{datastattable}.  The largest number of modules in the Turkey data is 107 whereas
the smallest number of modules in the NASA data is 161.  

There are a few important points about the Turkey data that distinguishes it from the data taken from the NASA projects.  The first
point is that the Turkey data was taken from a commercial appliance company.  This is must different from the NASA data in that the
software is made for commercial gain and in an environment that is completely unrelated to government projects.  The second point is that
this software was developed by people from a country other than the United States of America.  This fact creates a distinguishing divide
between the two types of projects.  Another point is that this software was created in an ad$-$hoc, informal method using groups of only
two or three people.  This is far less structured than the types of teams that developed the NASA data.  We believe that these facts about
the differences in the data, coupled with the results of \sect{turkeyresults}, show that the Which machine learner has some support
for working in a more universal way, as opposed to detectors that work on only one type of project.
\begin{table}[!ht]
\begin{center}
	\begin{tabular}{|r|c|c|c|}
		\hline
		\textbf{File Name} & \textbf{Type} & \textbf{Defects} & \textbf{Modules} \\ \hline \hline
		ar3 & Washing Machine & 8 & 63 \\ \hline
		ar4 & Dishwasher & 20 & 107 \\ \hline
		ar5 & Refrigerator & 8 & 35 \\
		\hline
	\end{tabular}
\caption{Information About the Turkey data.}
\label{tb:turkeyinfo}
\end{center}
\end{table}
\subsection{Experiment}
\label{turkeyexp}
This experiment was set up in a similar fashion to the experiment discussed in \sect{defectmdp}.  The same process of cross validation using a
$10x3$ method.  As with the MDP experiment, several different types of variants of Which were used and these are reported in \fig{turkeyqtc}
and \fig{turkeymwu}, but were not reported in \fig{turkeyplot} for the sake of clarity in the graphs.  The same variants of Which that are
discussed in \tab{whichvartab} were also used in this experiment.
\subsection{Results}
\label{turkeyresults}
Since the Turkey experiment only consisted of three seperate project files, we feel it is uncessary to catergorize in the same fashion as the 
MDP experiment in \sect{defectmdp}.  The same layout conventions are still used, however, that is there are three figures, \fig{turkeyplot},
\fig{turkeyqtc} and \fig{turkeymwu}, that all report results on the data.  An overall quartile chart and MannWhitney U$-$Test is also reported.

\fig{turkeyoverallqtc} and \fig{turkeyoverallmwu} represent the performance of each of the detectors over all three of the projects.  It gives insight to how well each
detector worked as a whole, instead of the individual projects that the other results show.  An important fact to notice here is that overall, manual$-$Up
was the best performer. A possible explaination of this involves the nature of the test.  The single project where manual$-$Up had the best overall performance
in the MannWhitney U$-$Test was ar3.  ar3 has a total of 63 modules with only 8 of them being defective.  Our cross$-$validation first randomizes
the data before splitting it into 3 train$-$test sets.  Each module has a 66\% chance to be in the train set and a 33\% chance to be in the test set.
A total of $63(0.33)=21$ modules are in each of the test sets.  The distribution of defective modules in the set is $\frac{8}{63}=0.126$.  If only
21 of the modules are in the test set and the distribution of defects is 12.6\%, then the number of defective modules, on average, in the test
set will be $21(0.126)=3$.  It is quite possible for those 3 defective modules per test set to have a small line of code attribute and thus be
explored first.  Since we are only dealing with very small sample sizes, manual$-$Up has an advantage of being able to quickly identify the
defective modules.  The problem with manual$-$Up is that it has no detector, it simply sorts the modules by the line of code attribute and explores
them all.  As mentioned in \sect{korudiagram}, the area under the curve measurement is the area under each of the curves plus the area of the rectangle
representing the horizontal line extension of the curve to a point $(100, y)$ on the Koru Diagram.  Since all of the remaining modules in the
test space are not defective, manual$-$Up will travel this space in a horizontal line, in essence emulating the oracle.  The oracle technically, stops
at the point $(x,100)$, however, and the straight line is artificially inserted into the space to guarentee that it will have the most area under
its curve.  So while the detectors generated from the machine learners perform \textit{slightly} worse than manual$-$Up, it is important to note
that they do not end at the point $(100,100)$, like manual$-$Up does.  This is only a possible explaination for this effect with the Turkey data, but
one that we think holds merit.

Another intersiting point that can be seen looking at this data is that all of the machine learners performed about the same according to both
the MannWhitney U$-$Tests and the Quartile charts.  This does give some merit to the classic learners of j48 and \nbc, but it also
gives more merit to Which.  The goal of the Turkey experiment was to show that Which can be used to create detectors over a broader range
of software projects.  The fact that it performed well compared to the other machine learners only helps that claim.  It is important
to note that while the classic machine learners did perform well on this data, they did not perform well on the NASA data.  This instability in
their performance is countered by the fact that Which performs well on both data sets.

A final point of interest that is seen in these results is that the other variants of Which, described in \tab{whichvartab}, also performed on par
with the classic learners.  For the reasons illustrated in the above paragraph, however, the instability over multiple project cases is undesirable.
It is interesting that our variants using the $h_{\frac{PD}{LOC}}$ heuristic performed well, however, because we had initially assumed the general
correlation of pd and effort should not work, given the discussion in \sect{defectwhichh}.
\begin{figure}[!ht]
\begin{center}
\begin{tabular}{cc}
	\resizebox{60mm}{!}{\includegraphics{figures/chapter4/Turkey/plots/ar3.png}} &
	\resizebox{60mm}{!}{\includegraphics{figures/chapter4/Turkey/plots/ar4.png}} \\
	ar3 & ar4 \\
	\resizebox{60mm}{!}{\includegraphics{figures/chapter4/Turkey/plots/ar5.png}} & \\
	ar5 & \\
\end{tabular}
\caption{Graphs representing the results for the Turkey experiment.}
\label{fig:turkeyplot}
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
	\input{turkeyqtc}
\caption{Quartile Charts Correlating to \fig{turkeyplot}.}
\label{fig:turkeyqtc}
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
	\input{turkeymwu}
\caption{MannWhitney U$-$Tests Correlating to \fig{turkeyplot}.}
\label{fig:turkeymwu}
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
	\input{turkeyoverallqtc}
\caption{Quartile Charts Representing the Overall Performance of the Detectors on the Turkey Experiment.}
\label{fig:turkeyoverallqtc}
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
	\input{turkeyoverallmwu}
\caption{MannWhitney U$-$Tests Representing the Overall Performance of the Detectors on the Turkey Experiment.}
\label{fig:turkeyoverallmwu}
\end{center}
\end{figure}

\clearpage

\section{Category III: \att Data}
\label{atntdata}
The following experiment was done using data from the \att Laboratory in New Jersey.  This data has been used in several other experiments
by Dr. Thomas Ostrand and Dr. Elaine Weyuker and Dr. Robert Bell\cite{ostrand1,paretosrc3}.
This data was proprietary so no statistics are reported about the data.
\subsection{Data}
\label{atntdatadesc}
The \att data is very different from the data used in the experiments of \sect{defectmdp} and \sect{turkeydata}.  Unlike the previous
experiments, this is the only data tested on that consists not of seperate projects but instead one project broken up into
$35$ releases.  Each release contains information relating to the previous releases. Also data does not use the Halstead\cite{halstead}
and McCabe\cite{mccabe} attributes that were present in those data sets.  In fact, individual instances in this data correspond to files
in the project, not modules.  This is a less fine$-$grained approach than the modules of the previous defect detection process.  A file
is different from a module in that, in C$++$ for example, a file may contain an entire class's implementation, thus containing many modules
per file.  Also, if a module is flagged in being defective is would be easier to test that module for the specific defect.  However, if an
entire class implementation file is flagged as defective it could require more extensive testing.  A very important difference with this
data compared to the previous MDP and Turkey data sets is that the defect class is not a boolean.  In this data, the class is numeric and
contains the number of defects found in that file.  This creates some problems with comparisons with our previous test because defective
instances are no longer equal.  That is if an instance is found to have $100$ defects, it should be much more important to detect over an instances
that has only $5$ defects.

\begin{table}[!ht]
\begin{center}
\begin{tabular}{|r|l|} \hline
	\textbf{Attribute} & \textbf{Description} \\ \hline \hline
	File Age & Describes how long this file has been in the project. \\ \hline
	Is  New? & Boolean attribute that is 1 if the file was created this project and\\
			 & 0 if it was created previously. \\ \hline
	New Developers & Number representing the new developers that have worked \\
				   & on this file since the previous release. \\ \hline
	New Developers $-1$ & Number representing the new developers that have \\
						& worked on this file in the previous release. \\ \hline
	New Developers $-2$ & Number representing the new developers that worked \\
						& on this file in the previous previous release. \\ \hline
	Cum Developers & Total number of developers that have worked on this file \\
				   & since its creation. \\ \hline
	Cum Developers $-1$ & Total number of developers that have worked on this \\
						& file since its creation until the previous release. \\ \hline
	Cum Developers $-2$ & Total number of developers that have workeed on this \\
						& file since its creation until the previous previous release. \\ \hline
	LOC & The number of lines of code that this file contains. \\ \hline
	Language & The programming language this file was written in. \\ \hline
	Defects & Number of defects this file contains. \\ \hline
\end{tabular}
\caption{Information About the Attributes Used in the \att Data Sets.}
\label{tb:attinfo}
\end{center}
\end{table}

\tab{attinfo} gives a description of each of the different attributes used in this data file.  This is fundamentally different from the other
experiment's data for a few reasons.  As mentioned previously,  the Halstead and McCabe metrics are not in this data about the instances.  Also,
there is no information at all about the actual source code besides the language it was written in and the amount of lines of code that each 
instance contains.  The idea behind most of these attributes is that it is the method the files were accessed and by whom they were accessed that
causes the defects in the instances.  It should be said that another idea about the statistical information that the creators of this thought
could cause more errors is the relation of modifications to this file that were done both one and two releases ago. It can be said that having
this previous and previous previous release information creates a situation where several attributes in the data set are highly correlated.  In
the above table, the defects attribute was changed from an integer to a boolean where it was given true if $defects \geq 0$ and false otherwise.

As a final note, we are under a nondisclosure agreement about the data we used in these experiments and are not allowed to publish any information
about the data that we haven't already.  This includes the number of files, names of files, and number of defects per file, distribtion of
attribtues as well as the actual data itself.
\subsection{Experiment}
\label{atntexp}
As was stated in \sect{atntdatadesc}, this data consists of 35 seperate releases and this created a convenient method of training and testing the
machine learners that removed the need for the $10x3$ cross$-$validation of the previous experiments.  Instead, we ran a series of tests using
releases $4$ through $35$as test data and as train data used the previous $5$ releases. For release tests $4$ and $5$, all available 
previous releases were used, since there were less than $5$ previous releases.  In other words, if we wanted to test the machiene learners
on release $7$, we would construct a train set consisting of releases $2-6$.  This is very equivalent to a real world situation since many
projects are deployed or at least tested in a release format.  A machine learner would have access to all of the previous releases to learn
from in order to detect defective files in the current release to be tested.  The reason we did not allow more than $5$ previous test releases
is due to time constraints, we only had $5$ days to use the data and some of these experiments could take a very long time to run.  However,
the experiments of \cite{ostrand1,paretosrc3}, used only the previous $n$ releases as well.

The original goal of this experiment was to compare our machine learner, Which, to the negative binomial regression used by Dr. Ostrand et al.
prevously on the data that reported good results.  More information on these experiments can be see in \sect{previousexp}.  However, as
was discussed in \sect{atntdatadesc}, their detector did not simply attempt to single out defective files but also rank the defective files
based on some predicted number of defects this file would have.  The current state of our tests so far have been done using the boolean class
of defective or not defective and the Koru Diagram of \sect{korudiagram} was meant for such a boolean class system.  Altering the data to
have a booleana class was done and as such a fair comparison could not be made with Which and the negative binomial regression.  This is
because we are ranking our learners based on number of defective instances singled out whereas their system was weighting which predicted
defective modules to list first.  Instead of comparing Which to the negative binomial regression, we instead compared the classic learners
of j48, Ripper, and \nbc with Which.  The manual detectors of manual$-$Up and manual$-$Down were also included in our studies.

The distributions of these attribute values were significantly different than those of \sect{defectmdp} and \sect{turkeydata}. So
to deal with this change in distribution, we use Which10 in these experiments instead of the Which2 that we have used prior.  Which10
is simply the ordinary Which process that uses $10$ bin equal frequency discretization as opposed to the $2$ bin equal frequency 
discretization that Which2 uses. Also, we have decided to include a micro$-$sampling varaint in these experiments as well.  
This shows up in the result figures as Which10010.  Which10010 uses a micro$-$sampling with $n=100$ and $10$ bins equal frequency
discretization.  The process of micro$-$sampling is described in detail in \sect{microsampling}.  The experiment of \sect{micro}
is an experiment dedicated to the effects of micro$-$sampling as a preprocessor with Which.  The reason we have included
a micro$-$sampling variant of Which in this experiment was due to time constaints.  We did not have enough time
to do an entire micro$-$sampling experiment on the \att data, so we opted to only include one variant of Which that does it.

\subsection{Results}
\label{atntresults}
In order to save space, only a few of the resulting graphs, quartile charts, and MannWhitney U$-$Tests are reported here, but the overall
results reported later in this section contain result information based on all $32$ total tests conducted.

\fig{atntplots}, \fig{atntqtc}, and \fig{atntmwu} give the plot, quartile chart, and MannWhitney U$-$Test results for $4$
different releases respectively.  The results here are typical results and were chosen because these four releases had more faulty
files than some of the other ones, making some of the notable points easier to see in the plots.  It can be seen here that the
oracle behaves in the same way it did in the experiments of \sect{defectmdp} and \sect{turkeydata}.  The oracle's line has a very
high slope implying that most of the faults are found in the smaller modules.  The plot of release four looks exactly like our
initial illustration of the Koru Diagram, \fig{koruexample}, found in \sect{korudiagram}.  

It is interesting to see the changes
that occur with the manual$-$Up and manual$-$Down detectors as the software project matures.  As was stated, release $4$ looks
like something we would expect, with manual$-$Up traversing the Koru space in a similar fashion to the oracle, even though the
slope is much less, and manual$-$Down traversing the Koru space in the opposite way.  However, in the later releases, namely
releases $19$ and $24$ reported here, manual$-$Up and manual$-$Down traverse the space nearly parallel with each other, and in release
$24$ manual$-$Down is above manual$-$Up.  The fact that these two detectors are parallel with each other implies that the underlying
distribution of faults in the project files is evenly distributed among large and small files.  However, looking at the oracle
suggests that this is not the case.  An explaination of this is that the files are all very similar in size, so the effect of
sorting them by their lines of code is lost here.  The fact that the oracle still rises very quickly to the $100\%$ pd line in
these plots can be explained by the fact that while all of the files are close to the same size, a small percentage of those
files actually contain faults.  In release $19$, $\frac{20}{579} \approx 3.4\%$ of the files actually contain errors.
Another test on this exact data was done by Ostrand et al. and they concluded that the
distribution of faults lies in the Pareto distribution described in \sect{pareto}\cite{paretosrc3}.  Their conclusion matches
the conclusion we have stated here that explains why the oracle still maintains a very large slope even though the file
sizes are all similar.

\begin{figure}[!ht]
\begin{center}
	\begin{tabular}{cc}
		\resizebox{60mm}{!}{\includegraphics{figures/chapter4/ATNT/plots/atnt4.png}} &
		\resizebox{60mm}{!}{\includegraphics{figures/chapter4/ATNT/plots/atnt19.png}} \\
		Release 4 & Release 19 \\
		\resizebox{60mm}{!}{\includegraphics{figures/chapter4/ATNT/plots/atnt24.png}} &
		\resizebox{60mm}{!}{\includegraphics{figures/chapter4/ATNT/plots/atnt30.png}} \\
		Release 24 & Release 30 \\
	\end{tabular}
\caption{Plots of Selected Results from \att Experiment.}
\label{fig:atntplots}
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
	\input{atntqtc}
\caption{$4$ Result Quartile Charts Taken from \att Experiment.}
\label{fig:atntqtc}
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
	\input{atntmwu}
\caption{$4$ Result MannWhitney U$-$Tests Taken from \att Experiment.}
\label{fig:atntmwu}
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
	\input{atntoverallqtc}
\caption{Overall Quartile Chart for the \att Experiment.}
\label{fig:atntoverallqtc}
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
	\input{atntoverallmwu}
\caption{Overall MannWhitney U$-$Tests Taken for the \att Experiment.}
\label{fig:atntoverallmwu}
\end{center}
\end{figure}

\fig{atntoverallqtc} and \fig{atntoverallmwu} show the overall quartile chart and Mann Whitney U$-$Test results for the \att
experiment respectively.  These results show that manual$-$Up is the overall winner over the $32$ total release based experiments
that were conducted.  This says that manually inspecting this code would be better than taking the advise of both the classic
machine learners of j48, Ripper, and \nbc as well as our new learner, Which. We have two possible explanations for these results 
conflicting with our results in the experiments of \sect{defectmdp} and \sect{turkeydata}.

The first of these explanations is that the attributes that were used to describe the \att data are vastly
different from the attribute used to describe the MDP and Turkey data.  While the MDP and Turkey data use attributes to describe specific
properties about the actual source code itself, the \att data uses attibutes to describe the methods the code was accessed and written over the
course of several releases.  An idea about these differences is that it is possible that the Halstead\cite{halstead} and McCabe\cite{mccabe}
statistics about the source code are easier to learn from and therefore create a performance decrease.

The second of these explanations was something we realized only after we had left the \att Laboratory.  As was stated before, we only had
one week to run our experiments on the \att data.  Our explanation about the behavior of the oracle remaining constant throughout all
three experiments but manual$-$Up and manual$-$Down differing greatly in the \att experiment from the MDP and Turkey experiments was that
the lines of code of each file is generally the same between all files in the release but the number of files that contain faults is still
in the minority.  This implies that the line of code attribute naturally is not exponential in distribution.  Also, looking at some of the
attribute types in \tab{attinfo}, it is highly probable that these types of attributes do not lie in an exponential distribution like
the attributes described in \tab{mdpdatadesctable}.  However, we had used the same rig to save time and that rig, by default, took the
natural logarithm of each attribute before the machine learners trained and tested on it.  The problem with exponentially distributed attribute
ranges is discussed in \sect{lognums}.  The same problems are seen when an attibute has a logarithmic distribution as well, except the bins
with the largest range will be the first few bins in discretization, and the remaining bins will all have very small ranges.  This is problematic
to learners for reasons discussed in \sect{lognums}.
\clearpage

\section{Category IV: Micro$-$Sampling}
\label{micro}
The process of micro$-$sampling is discussed in detail in \sect{microsampling}.  For this experiment, we decided to use micro$-$sampling with
Which2 to see if the concept of micro$-$sampling, which received good results in \cite{micro}, would work to Which's advantage on the experiments
of \sect{defectmdp} and \sect{turkeydata}.
\subsection{Experiment}
\label{microexp}
This experiment was done after the experiments using the classical learners and the various Which variants of \tab{whichvartab} were done.  Since we
concluded from the results of those experiments that Which2 was the best performer, these micro$-$sampling experiments will use Which2 as the 
variant of Which.  Also, for the purposes of clarity, and because it was beyond the scope of this experiment to do so, the classic machine
learners and the various Which variants were not included in this experiment.  As was explained in \sect{microsampling}, micro$-$sampling
is the process of creating a $50-50$ distribution of classes, as well as a specific number of instances, in data sets that contain two classes.
Since research has been done on micro$-$sampling and claims of this constant number being $\geq 25$ have been reported, we decided to use several
different sizes in this experiment to see how well Which performs with them. \tab{whichmicrovartab} provides an expaination of the various 
micro$-$sampling variants of Which2 used in this experiment.  Smaller versions of micro$-$sampling were used in this experiment, namely micro5
and micro10, but their results were too unstable and heavily dependant on the 10 and 20 random instances of the data that were chosen.  It is
because of this fact that they were not included in the results.
\begin{table}[!ht]
\begin{center}
	\begin{tabular}{|r|l|} \hline
		\textbf{Name} & \textbf{Description} \\ \hline \hline
		Which2 & A standard version of Which with $2$ bins equal frequency discretization. \\ \hline
		Micro20 & Micro$-$sampling using 20 true and 20 false instances. \\ \hline
		Micro30 & Micro$-$sampling using 30 true and 30 false instances. \\ \hline
		Micro50 & Micro$-$sampling using 50 true and 50 false instances. \\ \hline
		Micro75 & Micro$-$sampling using 75 true and 75 false instances. \\ \hline
		Micro100 & Micro$-$sampling using 100 true and 100 false instances. \\ \hline
	\end{tabular}
\caption{Description of Different Micro$-$Sampling Values used in \sect{microexp}.}
\label{tb:whichmicrovartab}
\end{center}
\end{table}
As with the experiments of \sect{defectmdp} and \sect{turkeydata}, this experiment used a $10x3$ cross$-$validation coupled with 
taking the natural logarithm of each of the attributes as a preprocessor to the experiment.  There were actually two experiments performed
with micro$-$sampling, the first was on the MDP data of and the second was on the Turkey appliance data.  These results were combined for
an overall look of the performance of the micro$-$sampled variants of Which and will be discussed in \sect{microresults}.
\subsection{Results}
\label{microresults}
\begin{figure}[!ht]
\begin{center}
	\input{microqtc}
\caption{Quartile Chart Results for Micro$-$sampling.}
\label{fig:microqtc}
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
	\input{micromwu}
\caption{MannWhitney U$-$Tests for Micro$-$sampling.}
\label{fig:micromwu}
\end{center}
\end{figure}

\fig{microqtc} and \fig{micromwu} contain the quartile charts and the MannWhitney U$-$Tests taken from the experiment on the NASA and the Turkey
data.  As with the MDP experiment, there are three distinct categories of results that can be gathered from this figures.  

The first category is where Which2 still outperforms its micro$-$sampled counterparts.  These occur on the files kc3mod and ar4.  
These victories are gathered by looking at the MannWhitney U$-$Tests of \fig{micromwu}.  By comparing the quartile charts to the MannWhitney 
U$-$Tests for kc3mod, it can be seen that while Which2 does have a better performance, the results are not nearly as standout as the results of
ar4.  

The second category is the category where most of the micro$-$sampled variants tie with Which2 and each other.  This result can be seen in cm1, 
kc2, kc3, pc1, mw1mod, ar3, and ar5.  This result is very interesting because the variant Micro20 is only dealing with a very small number of 
instances.  cm1 contains $49$ defective modules and $449$ other modules.  Micro20 will reduce that set to only $20$ of each type of module, 
effectively throwing away information about $429$ of the non$-$defective modules.  This is counter$-$intuitive because it \textit{should} be 
the case that the  information about the non$-$defective modules would help the Which learner to better decide what not to include in its rules.
This is the largest of the three categories containing $\frac{7}{10}=70\%$ of the total data sets that fall into this category.  This type of result
really leads to the conclusion that \cite{micro} said about micro$-$sampling not hurting the performance of machine learners.  More about the
conclusions is discussed further down in this section.

The third category is the category where Which2 loses to some or all of the micro$-$sampled variants of Which.  The data set that falls into this
category is mc2mod.  mc2mod is the data set that Which2, and all other Which variants, lost to all of the classic machine learners in the
MDP experiment of \sect{defectmdp}.  The fact that Which2 also lost to its micro$-$sampled variants in this file is interesting by comparison.
Inserting the results of Micro20 into the results of \fig{mdprescat3qtc} would place it below \nbc but above manual$-$Up.  mc2mod contains
$52$ defective modules out of $161$ total modules, according to \tab{nasainfo}.  Micro20 would reduce the defective modules to $\frac{20}{52}=38.4\%$
and the non$-$defective modules to $\frac{20}{109}=18.3\%$.  It is intersting to see that using micro$-$sampling, at least for mc2mod, drastically
improves the performance of the Which machine learner on this data set.  This is, however, only one instance where this type of standout result occured
and we are not claiming it as support for micro$-$sampling being an effective strategy for improving the performance of machine learners.

\begin{figure}[!ht]
\begin{center}
	\input{overallmicroqtc}
\caption{Overall Quartile Chart for Micro$-$Sampling.}
\label{fig:overallmicroqtc}
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
	\input{overallmicromwu}
\caption{Overall MannWhitney U$-$Tests for Micro$-$Sampling.}
\label{fig:overallmicromwu}
\end{center}
\end{figure}

\fig{overallmicroqtc} and \fig{overallmicromwu} show the overall performance comparison results for Which2 and the micro$-$sampled variants.  Counter
to our conclusion in the discussion above about the second category of the micro$-$sampling experimental results, Which2 has the best performance
compared to its micro$-$sampled variants.  The only variant that ties with Which2 is Micro20, according to the MannWhitney U$-$Tests.  However,
according to \fig{overallmicroqtc}, the means and quartiles of the variants are all very similiar.  While it is clear from the figure that Which2
has the best mean value, it is visible here that the magnitude of difference between the means and quartile distribution between the normal Which2
and the micro$-$sampled variance is much smaller.  The fact that Which2 tied with Micro20 is interesting because Micro20 reduces the data set
to a very small size. The data set pc1 contains $77$ defective modules and $1032$ non$-$defective modules.  This data set contains $1109$
modules in its normal state, yet the Micro20 Which variant, using a data set consisiting of only $40$ modules, performs just as well.

This experiment gives interesting insight to the effects of micro$-$sampling on defect detection data.  It tells us that removing both defective
and non$-$defective modules from a data set to produce a much smaller data set does not hurt nor help the performance of Which2.  Since micro$-$sampling
is a special form of undersampling, these results are not entirely unexpected.  This is an important result because it tells us that with limited
data for a project, it is still possible for effective defect detection.
