\begin{figure*}
\begin{center}
\begin{tabular}{ccc}
\includegraphics[angle=90,width=2in]{log/undercm1.pdf}&
\includegraphics[angle=90,width=2in]{log/underkc1.pdf}&
\includegraphics[angle=90,width=2in]{log/underkc2.pdf}\\
\includegraphics[angle=90,width=2in]{log/underkc3.pdf}&
\includegraphics[angle=90,width=2in]{log/undermc1.pdf}&
\includegraphics[angle=90,width=2in]{log/undermc2.pdf}\\
\includegraphics[angle=90,width=2in]{log/undermw1.pdf}&
\includegraphics[angle=90,width=2in]{log/underpc1.pdf}&
\includegraphics[angle=90,width=2in]{log/underpc2.pdf}\\
\includegraphics[angle=90,width=2in]{log/underpc3.pdf}&
\includegraphics[angle=90,width=2in]{log/underpc4.pdf}&
\includegraphics[angle=90,width=2in]{log/underpc5.pdf}
\end{tabular}
\end{center}
\caption{Results from Experiment \#3.}\label{fig:under}
\end{figure*}
\section{Experiment 3: Under-sampling WC}

Experiment \#2 was motivated by an observation
from the experiment \#1 results; i.e.
a small number of modules are sufficient for
defect prediction.

Experiment \#3 has a similar motivation. Recall from \fig{inc} that in majority case,
the performance rarely improved after training on just a  few hundred
examples.
Consider a 90\%/10\% train/test split on 100 modules $pc1$ modules.
\fig{data} tells us that  these 100 modules contain 6.94 defective modules (on average)
so the 90\% training set only contains around six defective examples.

This observation prompted an {\em under-sampling}~\cite{drummond03} experiment that explored
if very small example sets
perform as well larger example sets.
In under-sampling, the frequencies of target classes in a training set are adjusted
such that the minority class becomes more frequent. Under-sampling has many advantages
including reducing the computational cost of the learning (since fewer examples
are processed). Also, learning from fewer examples means learning from
fewer irrelevancies so, in theory anyway, the resulting predictor may be simpler~\cite{drummond03}.

Experiment \#3 had nearly the same experimental design as experiment \#2 (same learner,
same data pre-processing, same 12 tables of data) but treatment 3 was replaced with:
\bi
\item Treatment 4 (under-sampling WC):
In the training sets, a random selection on non-defective modules were discarded.
These discards continued until a new training set was generated containing an
equal number of defective and non-defective modules.
\ei
After this under-sampling, training sets were built using just the first 25,50,75,..$max$
examples
(selected at random). In order to preserve under-sampling, $max$ was capped at
twice the number of defective modules in the table of data.

Note that while we trained on under-sampled data,
we tested the learned predictor on
the {\em original} distribution of defective modules (see column 
$\%defective$ column in \fig{data}).
That is, this experimental design takes care to preserve the natural properties
of the test set.

An alternative to under-sampling is {\em over-sampling}
where
randomly selected examples of the minority class are copied. This paper does not
explore over-sampling since we are interested in {\em reducing} the overheads
with WC data collection. Also, Drummond \& Holte are very negative regarding
the benefits of over-sampling~\cite{drummond03}. Lastly, in one study with WC learning
on all the tables of \fig{data}, we found that over-sampling {\em reduced}
the performance of our preferred learner (a Bayes classifier)~\cite{gay07}.

\subsection{Results from Experiment \#3}

The results of  experiment \#3 follow the same pattern as seen above.
As shown in \fig{under},
 after a few dozen examples, there is little extra value in using
larger
training sets.

Mann-Whitney tests were applied to test this visual pattern. 
As before, the issue was
``how much data is enough?''
 i.e.
what is the $min$imum
training set size that never lost to other training set of a larger size. 
In a result that echoes experiment \#2, 
the
$min$ value 
was usually quite small:

\bi
\item
For $\{cm1,kc2,kc3,mc1,mc2,mw1,pc1,pc2\}$, $min=25$.
\item
For $\{pc3\}$ $min=75$, but predictors learned from 25 examples
only lost three times out of 11.
\item
For $pc4$ , $min=200$, but predictors learned from 25 examples
only lost once out of 13 trials.
\ei

In other tables of data, $min$ was much larger. In $\{kc1,pc5\}$ the
$min$ values were found at $\{575,1025\}$,  respectively.
However, much small training set sizes performed {\em nearly} as well
as these larger sets:
\bi
\item
In $kc1$, predictors learned from 25 examples only
lost to 575 examples once in 25 trials;
\item
In $pc5$, predictors learned from 25 examples only lost to 
1025 examples once in 25 trials.
\ei
\subsection{Discussion of Experiment \#3}
In the majority case, predictors learned from very small
under-sampled data 
perform nearly as well as predictors learned from many more examples.
At first glance,it may seem quite remarkable
that such small training sets (25 to 75 examples)
works so well. However, on reflection, the results of \fig{under} are not surprising:
\bi
\item
Under-sampling 
{\em increases} the frequency of the target class.
Repeating the $pc1$ example, observe that
an under-sampled $pc1$ table with 25 examples contains 12 or 13 defective
modules; i.e. {\em twice} as many examples as 100 randomly selected 
samples from the unsampled $pc1$ table.
Hence, it is not surprising that 25 under-sampled examples from \fig{under} does
as well larger unsampled examples.
\item
All these experiments use very simplistic static code measures such as lines of code, number of
unique symbols in the module, etc.  
Such simple measures have limited information content. Limited content
is soon exhausted by repeated sampling.
Hence, such simple measures reveal all they can reveal after
a small sample (just  a few dozen examples).
\ei
