Clump is being used as a software defect predictor
for the NASA/MDP data sets\footnote{The datasets can be obtained
at \url{http://promisedata.org}} from the Promise Data
Repository\cite{PromiseData}. The NASA/MDP datasets used are: CM1,
KC1, KC2, KC3, MC2, MW1, and PC1. These datasets were chosen for
examination because they represent a wide variety of projects. The NASA
datasets represent projects such as flight software, storage
management, video guidance systems, and an experiment
framework\cite{08Turhan}. NASA utilizes sub contractors for the
majority of its projects, so the seven projects represented in the MDP
datasets represent the work of many different development teams.

The files have been modified from their original form to allow them
to be used in cross company experiments\footnote{Cross company
experiments use 6 of the 7 datasets for training, and the $7^{th}$ for
testing}. They have been modified to unify the columns used, and their
orderings. The data is stored in the arff format for easy integration
with the WEKA experimenter\cite{weka}. 

Two different types of experiments are being run. First, Within
Company, and second, Cross Company. Within Company trains the learning algorithms with a large portion of data from one data set, and tests the learned theories with the remainder of the data from that dataset. Cross Company analysis trains the learning algorithms using all but one of the available data sets, and tests the theories on the remaining whole dataset. 

The Within Company data sets do a 5 by 5 way cross validation. This means that the instances are randomly ordered 5 times, and broken into 5 separate sets for each random ordering. Then 4 out of the 5 separate sets are used for training, and the remainder is used for testing in a round robin fashion. 

The Cross Company experiments do a 7 way cross validation. The data is randomly ordered in the test dataset and in the combined training datasets 7 times, with each data set being the test dataset once.

The experiment compares several variations of Clump to other learners,
mostly other rule tree learners: oner, naive bayes, j48, jRipper,
Ridor, and locally weighted learning(LWL). The other learners come
from Weka's Data Mining Software\cite{weka}. j48, jRipper, Ridor, and
Clump represent rule tree methods. OneR represents a rule based method
as well, but is not built on a rule tree. Naive Bayes and LWL are
frequency count methods that are based on Bayes
theorem\cite{bayestheory}. All experiments are run using
Ourmine\cite{ourmine}, a shell environment for data mining experiments
created at West Virginia University.
\input{fig/experimentPseudoCode}

\input{fig/numberOfRules}
\input{fig/numberOfConditions}

\subsection{Parameters to Clump}
Clump takes five different parameters:
\bi
\item Test file
\item Training file
\item Maximum level of mixedupedness
\item Number of bins for discretization
\item Split size
\ei
The test and training files are in the arff format described in Weka's
documentation\cite{weka}. Each file has a header where the column
information is included. The name of the data set, the names of the
columns, and the type of column\footnote{The columns can be either
continuous or discrete} is included. The body of the datafile contains
a comma separated list of all instances in the dataset.

The maximum level of mixedupedness represents a threshold on the
frequency of training instances not of the majority class allowed in a
node before a new rule is created. This also effectively puts a lower
bounds on the size of a node. A node can have no less than one more
than twice the level of mixedupedness training instances.

The number of bins for discretization is used when creating the
rules. The minimum and maximum values for a given attribute are
taken. The range is then broken up into n equal width bins according
to its value. More bins means a higher granularity in the rules
created, but it also means a lower number of training instance in each
node. Although it is often desired to reduce the number of instances
in each cluster, there exists a lower bounds where too few instances
can prevent new rules from being learned. The new rules could be
prevented because the number of instances in each rule could violate
the level of mixedupedness previously set.

The final parameter to Clump is the split size, or the number of
testing instances to consider at one time. The chunk of testing
instance being considered are placed in the rule tree. The training
data stored in each node of the rule tree where a testing instance
stops at is combined to form the training set for that chunk. This
training set and the subset of the testing set is then passed to a
Naive Bayes classifier for final classification. A higher split size
will increase the number of training instances used in classification,
and may add noise to the training set. Conversely, a small split size
can gather too little data to make an accurate prediction.

\subsection{Clump Configuration Values}
Optimal values for the split size, number of bins, and level of
mixedupedness were determined experimentally.  Any value greater than
or equal to 1 for the split size produced the same PD and PF
results. The value with the largest impact on the performance of Clump
was the number of bins used during discretization. A number of bins
ranging between 2 and 10 were tested, and as the bin size chosen
neared either extreme, the number of rules and runtime increased. A
bin count of 3-5 produced the lowest run-time and rule counts. Lower
bin counts also produce higher PD's and lower PF's. A split size of 5,
minimum amount of mixupedness of 10, and 5 bins were chosen for
further investigation.

\subsection{Experiments}
There were two main experiments run: the Within-Company tests, and the
Cross-Company tests. Each test was run twice. The first variation was
run with the data in its raw form, and the second variation took the
log of the numeric attributes before
classification. \fig{results-WC}, \fig{results-CC}, \fig{results-WC-Log},
and \fig{results-CC-Log} shows the results for the four different
tests.
