\section{Commissioning a Learner through Incremental Random Sampling}


OURMINE's use is not only important to researchers, but also to practitioners. 
To demonstrate this, two very simple experiments were conducted using seven PROMISE data sets  (CM1, KC1, KC2, KC3, MC2, MW1, PC1).
The aim of this experiment is to commission a data mining system for a local site.
When selecting the right data miners for a particular source of data, three issues are:
\be
\item What learners to use?
\item What discretizers to use?
\item How much data is required for adequate learning?
\ee
Given software being released in several stages, OURMINE can be used on stage 1 data to find the right answers to the above
questions. These answers can be applied on stages 2,3,4, etc.

In Part A of this experiment, four learners (Naive Bayes, J48, ADTree,
One-R) are trained using undiscretized data, and then the same
learners are trained on discretized data for a total of eight
combinations. While this represents a highly simplified proof-of-concept,
from it we can illustrate how our learners can be integrated with
other techniques to find the best treatment for the data set(s) of
concern. Practitioners can then short-circuit their experiments by
only performing further analyses on methods deemed worthwhile for
the current corpus. In turn, this decreases the amount of time in
which results can be obtained.

In Part B of the experiment, the winning treatment from Part A is
used with randomly selected, incremental sizes of training data.
In this experiment, it can be shown how early our quality predictors
can be applied based on the smallest number of training examples
required to learn an adequate theory. Predictors were trained using
$N$ = 100, $N$ = 200, $N$ = 300... randomly selected instances up
to $N$ = 1000. For each training set size $N$, 10 experiments were
conducted using $|Train|$ = 90\% * $N$, and $|Test|$ = 100. Both
$Train$ and $Test$ were selected at random for each experiment. Our
random selection process is modeled after an experiment conducted
by Menzies et. al.~\cite{me07i}. In that paper, it was found that
performance changed little regardless of whether fewer instances
were selected (e.g. 100), or much larger sizes on the order of
several thousand. Additionally, it was determined that larger
training sizes were actually a disadvantage, as variance increased
with increasing training set sizes.

\subsubsection{Results}

\fig{pds2} and \fig{pfs2} show the results from Part A of the experiment. As can be seen, ADTrees (Alternating Decision Trees) ~\cite{Freund99thealternating} trained on discretized data yields the best results: 
\\
\bi
\item{Mann-Whitney test ranking (95\% confidence): ADTrees + discretization provides the highest rank for both PD and PF results}
\item{Medians \& Variance: ADTrees + discretization results in the highest median/lowest variance for PD, and the lowest variance for PF, with only a 1\% increase over the lowest median} 
\ei


Results from Part B of the experiment are shown in \fig{pds3} and
\fig{pfs3}. Using the winning method from Part A, or learning using
ADTrees on discretized data, results show that unquestionably a
training size of $N=600$ instances is the best number to train our
defect predictors using the aforementioned experiment setup. This
is clear when considering $N=600$ maintains the highest Mann-Whitney
ranking (again using 95\% percent confidence), as well as the highest
PD and lowest PF medians.

More importantly, perhaps, it should be noted that while the most
beneficial number of training instances remains 600, we can still
consider a significantly smaller value of, say, $N=300$ without
much loss in performance; a training size of just 300 yields a loss
in PD ranking of only one, with a 3\% decrease in median, while PF
ranking is identical to our winning size and sporting medians of a
mere 3\% increase.

\begin{figure}[h]
\small
\begin{center}
\begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
\multicolumn{2}{c}{~}&\multicolumn{3}{c}{pd}&2nd quartile\\
\multicolumn{2}{c}{~}&\multicolumn{3}{c}{percentiles}&median,\\\cline{3-5}
Rank&Treatment& 25\%& 50\% & 75\%&3rd quartile\\\hline
1&J48 + discretization& 20& 66& 99&  \boxplot{0}{20}{66}{99}{7}  \\
1&ADTree + discretization& 25& 66& 97&  \boxplot{0}{25}{66}{97}{7}  \\
2&NB + discretization& 63& 73& 82&  \boxplot{0}{63}{73}{82}{7}   \\
2&One-R + discretization& 13& 56& 99&  \boxplot{0}{13}{56}{99}{7}  \\
3&J48& 29& 82& 96& \boxplot{0}{29}{82}{96}{7} \\
3&NB& 40& 79& 91& \boxplot{0}{40}{79}{91}{7}  \\
4&ADTree& 21& 83& 97& \boxplot{0}{21}{83}{97}{7} \\
4&One-R& 17& 83& 97& \boxplot{0}{17}{83}{97}{7} \\\hline
\multicolumn{5}{c}{~}&~~~~~0~~~~~~~~50~~~~100
\end{tabular}
\end{center}
\caption{Experiment \#1 - Part A - (Learner tuning). Probability of Detection (PD) results, sorted by rank then median values.  }\label{fig:pds2}
\end{figure}


\begin{figure}[h]
\small
\begin{center}
\begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
\multicolumn{2}{c}{~}&\multicolumn{3}{c}{pf}&2nd quartile\\
\multicolumn{2}{c}{~}&\multicolumn{3}{c}{percentiles}&median,\\\cline{3-5}
Rank&Treatment& 25\%& 50\% & 75\%&3rd quartile\\\hline
1&One-R + discretization& 0& 10& 86& \boxplot{0}{0}{10}{86}{7}  \\
1&ADTree + discretization& 2& 11& 75& \boxplot{0}{2}{11}{75}{7} \\
1&J48 + discretization& 1& 11& 80& \boxplot{0}{1}{11}{80}{7} \\
2&NB& 9& 19& 60&  \boxplot{0}{9}{19}{60}{7}  \\
2&J48& 4& 33& 72& \boxplot{0}{4}{33}{72}{7} \\
3&NB + discretization& 12& 25& 33&  \boxplot{0}{12}{25}{33}{7}   \\
3&ADTree& 3& 37& 79&  \boxplot{0}{3}{37}{79}{7}  \\
4&One-R& 3& 44& 83&  \boxplot{0}{3}{44}{83}{7}  \\\hline
\multicolumn{5}{c}{~}&~~~~~0~~~~~~~~50~~~~100
\end{tabular}
\end{center}
\caption{Experiment \#1 - Part A - (Learner tuning). Probability of False Alarm (PF) results, sorted by rank then median values.  }\label{fig:pfs2}
\end{figure}

\newpage

\begin{figure}[h]
\small
\begin{center}
\begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
\multicolumn{2}{c}{~}&\multicolumn{3}{c}{pd}&2nd quartile\\
\multicolumn{2}{c}{~}&\multicolumn{3}{c}{percentiles}&median,\\\cline{3-5}
Rank&Training Size& 25\%& 50\% & 75\%&3rd quartile\\\hline
1&600& 32& 88& 99&  \boxplot{0}{32}{88}{99}{7}  \\
1&700& 30& 86& 98&  \boxplot{0}{30}{86}{98}{7}   \\
1&900& 32& 86& 98&  \boxplot{0}{32}{86}{98}{7}  \\
1&800& 31& 85& 98&  \boxplot{0}{31}{85}{98}{7}  \\
1&500& 30& 84& 99& \boxplot{0}{30}{84}{99}{7} \\
1&1000& 30& 84& 99& \boxplot{0}{30}{84}{99}{7}  \\
2&300& 31& 85& 98& \boxplot{0}{31}{85}{98}{7} \\ 
2&400& 29& 84& 99& \boxplot{0}{29}{84}{99}{7} \\
3&200& 29& 83& 98& \boxplot{0}{29}{83}{98}{7} \\
3&100& 29& 77& 96& \boxplot{0}{29}{77}{96}{7} \\\hline
\multicolumn{5}{c}{~}&~~~~~0~~~~~~~~50~~~~100
\end{tabular}
\end{center}
\caption{Experiment \#1 - Part B - (Random Sampling). Probability of Detection (PD) results, sorted by rank then median values.  }\label{fig:pds3}
\end{figure}
\begin{figure}[h]
\small
\begin{center}
\begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
\multicolumn{2}{c}{~}&\multicolumn{3}{c}{pf}&2nd quartile\\
\multicolumn{2}{c}{~}&\multicolumn{3}{c}{percentiles}&median,\\\cline{3-5}
Rank&Training Size& 25\%& 50\% & 75\%&3rd quartile\\\hline
1&600& 1& 12& 68&  \boxplot{0}{1}{12}{68}{7}  \\
1&700& 1& 14& 70&  \boxplot{0}{1}{14}{70}{7}  \\
1&900& 1& 14& 68& \boxplot{0}{1}{14}{68}{7}  \\
1&1000& 1& 14& 69&  \boxplot{0}{1}{14}{69}{7}  \\
1&500& 1& 15& 69&  \boxplot{0}{1}{15}{69}{7}   \\
1&800& 1& 15& 69& \boxplot{0}{1}{15}{69}{7} \\
1&300& 2& 15& 69& \boxplot{0}{2}{15}{69}{7} \\ 
1&400& 1& 15& 71& \boxplot{0}{1}{15}{71}{7} \\
2&200& 2& 17& 71& \boxplot{0}{2}{17}{71}{7} \\
3&100& 4& 23& 71& \boxplot{0}{4}{23}{71}{7} \\\hline
\multicolumn{5}{c}{~}&~~~~~0~~~~~~~~50~~~~100
\end{tabular}
\end{center}
\caption{Experiment \#1 - Part B - (Random Sampling). Probability of False Alarm (PF) results, sorted by rank then median values.  }\label{fig:pfs3}
\end{figure}

