\section{Experiment \#2: Incremental WC }
\begin{figure*}
\begin{center}
\begin{tabular}{ccc}
\includegraphics[angle=90,width=2in]{log/incrementalcm1.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalkc1.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalkc2.pdf}\\
\includegraphics[angle=90,width=2in]{log/incrementalkc3.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalmc1.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalmc2.pdf}\\
\includegraphics[angle=90,width=2in]{log/incrementalmw1.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalpc1.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalpc2.pdf}\\
\includegraphics[angle=90,width=2in]{log/incrementalpc3.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalpc4.pdf}&
\includegraphics[angle=90,width=2in]{log/incrementalpc5.pdf}
\end{tabular}
\end{center}
\caption{Results from experiment \#2. Training set size grows in units of 100 examples,
 moving left to right
over the x-axis. The MC2 results only appear at the maximum x-value since MC2 only
contains less than 200 examples.
}\label{fig:inc}
\end{figure*}



\subsection{Design}
A curious aspect of the above results is that defect predictors
were learned using only a handful of defective modules.
For example, consider a 90\%/10\% train/test split on
$pc1$ with 1,109 modules, only 6.94\% of which are defective.
On average, the test set will only contain \mbox{$1109 * 0.9 * 6.94/100=69$}
defective modules. Despite this, $pc1$ yields an adequate median $\{pd,pf\}$ results
of $\{88,34.5\}$\%. 

Experiment \#2 was therefore designed to check how {\em little} data is required
to learn defect predictors. Experiment \#2 repeated the procedure of experiment \#1
without treatment \#1 (the
cross-company study). Instead,
experiment \#2
took the 12 example tables of \fig{data}
and 
learned predictors using:
\bi
\item Treatment 3 (reduced WC):  a randomly selected
subset of the 90\% rows of this table;
\ei
Specially, after randomizing the order of the rows, training sets
were built using just the first 100,200,300,\ldots rows in the tables.
After training, the learned theory was applied to  100 rows
not used in training (selected at random).

%:wXXX how many runs in all?

\subsection{Results from Experiment \#2}
As shown in \fig{inc}, there was very little change in balanced 
performance (from \eq{bal}) after learning from 100,200,300,... examples.
Indeed, there is some evidence that
learning from larger training sets had detrimental effects: the more
training data, the larger the variance in the performance of the learned detector.
Observe how, in $pc2$, as the training set size increases (moving right along the x-axis)
the dots showing the on balance performance start spreading out.
A similar, but smaller, {\em spread effect} can be see in $kc2$ and $mc1$.

The Mann-Whitney U test was applied to check the visual trends seen in \fig{inc}.
For each table, all results from training sets of size 100,200,300\ldots were
compared to all other results from the same table. At issue was ``how much data is enough?''
 i.e.
what is the $min$imum
training set size that never lost to other training set of a larger size. Usually, that 
$min$ value 
was quite small:

\bi
\item In seven tables $\{cm1,kc2,kc3,mw1,pc3,pc4\}$, $min=100$.
\item In  \{$kc1,pc1$\}, $min=\{200,300\}$ instances, respectively.
\ei

In other tables of data, $min$ was was much larger. In $\{pc2,mc1,pc5\}$ the 
$min$ values were found at $\{4900, 8300, 11000\}$, respectively.  However, much smaller
training set sizes performed {\em nearly} as well as these larger training sets:
\bi
\item In $pc2$, predictors learned from 800 examples only
lost to other size twice out of 53 trials;
\item
In $mc1$, predictors learned from 400 examples only lost to other sizes
once out of 92 trials);
\item
In $pc5$, predictors learned from 300 examples only lost to other sizes 
twice in 169 trials.
\ei
\subsection{Discussion of Experiment \#2}
In the majority case, predictors learned from just a few hundred
examples perform as well as predictors learned from many more examples.
This suggests that the effort associated with learning defect predictors
from within-company data may not be overly large.


