
\subsection{Analysis 3}

\begin{figure}[!h]
~\hrule~
{\scriptsize
\begin{verbatim}
an3() {
   local me="promise_an3"
   local bins=10
   local repeats=10
   local learners="nb"
   local datas="PC1 CM1 KC1 KC2 KC3 MW1 MC2"
   cd $Tmp
   (echo "#data,repeat,bin,increment,learner,goal,
   a,b,c,d,acc,pd,pf,prec,bal"
   for((r=1;r<=$repeats;r++)); do
      for data in $datas; do
         arff=$OURMINE/lib/arffs/mdp/shared_$data.arff
         blab "data=$data repeat=$r "
         seed=$RANDOM;

         if [ $data == "PC1" ]; then
            local increment="100 200 300 400 500 600 700 \
                             800 900 1500"
         elif [ $data == "CM1" ]; then
            local increment="100 200 300 900"
         elif [ $data == "KC1" ]; then
            local increment="100 200 300 400 500 600 700 \
                             800 900 1000 1100 1200 1300 \
                             1400 1500 1600 1700 1800 1900 2500"
         elif [ $data == "KC2" ]; then
            local increment="100 200 300 900"
         elif [ $data == "KC3" ]; then
            local increment="100 200 500"
         elif [ $data == "MW1" ]; then
            local increment="100 200 450"
         elif [ $data == "MC2" ]; then
            local increment="200"
         fi
         for((bin=1;bin<=$bins;bin++)); do
            blab "$bin"
            cat $arff | 
            someArff --seed $seed --bins $bins --bin $bin 
            goals=`cat $arff | classes --brief`
            for learner in $learners; do
               for i in $increment; do
                  blab "($i)"
                  buildIncrementalSet train.arff $i > inc.arff
                  $learner inc.arff test.arff | 
                  gotwant > results.dat 
                  for goal in $goals; do
                     cat results.dat |
                     abcd --goal "$goal" \
                     --prefix "$data,$r,$bin,$i,$learner,$goal" \
                     --decimals 1  
                   done
                done
             done                            
          done 
                        
          blabln 
       done
   done
) | sort -t, -n -k 15,15 | malign > $me.csv #each bin
blabln " "
echo ""; cat $me.csv
cp $me.csv $Safe/$me.csv
cd $Here
}
\end{verbatim}}
~\hrule~
\caption{Source Code for Analysis 3}
\label{fig:an3}
\end{figure}

The first two experiments looked at whether CC data models are a valid replacement for WC models. 
Consistently, it has been found that WC data is better if available. This analysis is concerned with
exactly how much local data is actually neccesary. Can defect predictors be built from small
amounts of local data?

One important note about the first two analyses - defect predictors were learned
using only a handful of defective modules. For example, consider the NASA set PC1. 
This set contains 1,109 modules, only 6.94\% of which are defective. This set is split into
a training set with 90\% of modules and a testing set with the remaining 10\%. 
On average, the training set will only contain $1109 \cdot 0.9 \cdot 6.94/100 = 69$ defective modules. 
Despite this, these NASA sets yield an adequate median \{pd, pf\} results of \{68, 31\}\%.
    
This analysis has been designed to find the smallest amount of data needed to learn
defect predictors. The full source code can be seen in \fig{an3}. Essentially, this experiment
uses the same tests as the first, but without the cross-company data. 
These defect predictors are built using reduced WC data: a subset of each project built in increasing increments.
Training sets were built using just the first 100, 200, 300, . . . data samples in the project. 
After training the defect predictor, its performance is tested on the remaining data samples not used in training.

\subsection{Results for Analysis 3}

\begin{figure}[!t]
\scriptsize
\centering
\begin{tabular}{c|rr}
Data Set & Incriment & Tie-Win-Loss\\\hline
 CM1 & All &    3,   0,    0 \\
 & 300 &    3,   0,    0 \\
 & 200 &  3,   0,    0\\
 & 100 &    3,   0,    0 \\
\hline
  PC1&900&    8,   1,    0\\
  &800&    8,   1,    0\\
  &700&    8,   1,    0\\
  &600&    8,   1,    0\\
  &500&    8,   1,    0\\
  &400&    8,   1,    0\\
  &200&    8,   1,    0\\
  &300&    9,   0,    0\\
 &All&    9,   0,    0\\
  &100&    2,   0,    7\\
\hline
 KC1&100&    1,  18,    0\\
  &200&   10,   9,   0\\
  &800&   18,   0,    1\\
  &600&   18,   0,    1\\
  &500&   18,   0,    1\\
  &400&   18,   0,    1\\
 &All&   18,   0,    1\\
  &900&   17,   0,    2\\
  &700&   17,   0,    2\\
  &300&   17,   0,    2\\
 &1000&   17,   0,    2\\
\hline
KC2&100&    1,   2,    0\\
 &200&    2,   1,    0\\
 &300&    1,   1,    1\\
 &All&    0,   0,    3\\
\hline
 KC3&All&    1,   1,    0\\
 &200&    1,   1,    0\\
 &100&    0,   0,    2\\\hline
 MW1&All&    2,   0,    0\\
 &200&    2,   0,    0\\
 &100&    2,   0,    0\\
\end{tabular}
\caption{Mann-Whitney Results} 
\label{fig:incResults}
\end{figure}


Recall \fig{measures}, "Balance" is defined to be a combination of \{pd, pf\} that decreases if pd decreases
or pf increases. This was used as the performance measurement for this analysis.  
As shown in \fig{incResults}, there was very little change in balanced performance
as the training set increased in size. Our results are practically the same as the original paper's in this regard. 

The Mann-Whitney U test was applied to check the statistical difference in the results. For each project, the results from
each size increment were compared to all other increments from the same project. The question is "how much data is enough?"
This is answered by finding the smallest set that never loses to a larger set. At mimimum, 
there is no difference between smaller sets and larger sets. In fact, there is some evidence that learning
from larger training sets is worse than using a smaller set. On set PC1, the largest training set is one of the worst.
The increased noise causes a drop in the balance. The best results for set KC2 are the two smallest increments. In general, 
the minimum number of examples needed is quite small: 
\bi
\item In data sets \{CM1, KC1, KC2,MW1\}, min = 100;
\item In \{PC1,KC3\}, min = 200;
\item Set MC2 has fewer than 200 modules, therefore there is no reason to test it. 
\ei

We explain the results of this analysis as follows. These data sets use simplistic static code
features such as lines of code, number of unique symbols in the module, etc. 
While these static code features are useful for certain types of analysis, they do not provide
a complete picture of a function's internal structure. 
These static code features may only contain limited information content ~\cite{menziesgay08}.
Such limited content is soon exhausted by repeated sampling. These simple features reveal
all they can reveal after a small sample, and additional sampling only introduces noise and variance. 


