\subsection{Analysis 1}

\begin{figure}[!h]
~\hrule~
{\scriptsize
\begin{verbatim}
an1() {
   local me="promise_an1"
   local bins=10
   local repeats=10
   local learners="nb"
   local datas="PC1 CM1 KC1 KC2 KC3 MW1 MC2"
   cd $Tmp
   (echo "#data,repeat,bin,treatment,learner,goal,a,b,c,d \
      ,acc,pd,pf,prec,bal"
   for((r=1;r<=$repeats;r++)); do
      for data in $datas; do
         arff=$OURMINE/lib/arffs/mdp/shared_$data.arff
         combined=$OURMINE/lib/arffs/mdp/combined_$data.arff
         blab "data=$data repeat=$r "
         seed=$RANDOM;
         for((bin=1;bin<=$bins;bin++)); do
            blab "$bin"
            cat $arff | 
            someArff --seed $seed --bins $bins --bin $bin 
            goals=`cat $arff | classes --brief`
            for learner in $learners; do
               $learner train.arff test.arff | 
               gotwant > results.dat 
               for goal in $goals; do
                  cat results.dat |
                  abcd --goal "$goal" \
                  --prefix "$data,$r,$bin,WC,$learner,$goal" \
                  --decimals 1  
               done
            done                            
            goals=`cat $arff | classes --brief`
            blab "CC"
            local N=$[ ( $RANDOM % 10 ) +1 ]
            blab "($N)"
            cat $arff | 
            someArff --seed $seed --bins $bins --bin $N
            for learner in $learners; do
               makeTrainCombined $combined > trainCom.arff
               $learner trainCom.arff test.arff | 
               gotwant > results.dat 
               for goal in $goals; do
                  cat results.dat |
                  abcd --goal "$goal" \
                  --prefix "$data,$r,CC,CC,$learner,$goal" \
                  --decimals 1  
                done
             done  
          done 
          blabln 
       done
    done       
 ) | sort -t, -n -k 12,12 | malign > $me.csv 
 blabln " "
 echo ""; cat $me.csv
 cp $me.csv $Safe/$me.csv
 cd $Here
}
\end{verbatim}}
~\hrule~
\caption{Source Code for Analysis 1}
\label{fig:an1}
\end{figure}

The goal of the first experiment is to determine whether using cross company data 
has any benefit for defect prediction and, if so, the conditions under which cross-company data 
should be preferred to within-company data. Our first analysis uses the seven NASA MDP projects and follows
the source code given in \fig{an1}. 

These NASA projects come from different sources, and certain sets contain different attributes. 
This could be a potential problem for the classifier to be used. For this analysis, only the set
of 19 common attributes was used and extraneous columns were removed from each data set. 

For each project, test sets were built from 10\% of the data, selected at random. 
The training sets used by Naive Bayes were build from:
\bi
\item CC data: The other six NASA projects are combined, then 90\% of "true" and 90\% of "false" instances are used.
\item WC data: remaining 90\% data of the current project;
\ei
Naive Bayes classifiers were used since previously, we have found them to generate better median
performance that alternate methods. Also, as shown by the results of Lessman et al.~\cite{me07b}, their performance is
not statistically different from a large number of supposedly more sophisticated systems~\cite{lessmann09}.

As shown in \fig{an1}, a full 10x10 cross-validation is performed in order to control for order effects (where the learner is affected by
the order of examples). Each time, the data is randomized and different bins are chosen for the training and test sets. These results are passed to
the abcd function, which calculates statistics such as probability of detection, probability of false alarm, and precision.  
The thing to note from \fig{an1} is that, twice we call
\begin{center}\scriptsize
\verb+ for learner in $learners; do+
\end{center}
The first time generates the WC results since the train and test sets come from the same data sets. The second
time generates the CC results using the same test set but building an alternate training set from all the other
data sets. 


\begin{figure}[!h]
\centering\scriptsize
\begin{tabular}{l|c|rrr}
Data Set & Treatment & Tie&Win&Loss\\\hline
CM1 & WC &    0&   1&    0 \\
CM1 & CC &    0&   0&    1 \\
&&&&\\
PC1&WC &    0&   1&    0\\
PC1& CC&    0&   0&    1\\
&&&&\\
KC1&CC &    0&   1&    0\\
KC1& WC&    0&   0&    1\\
&&&&\\
KC2&WC &    1&   0&    0\\
KC2& CC&    1&   0&    0\\
&&&&\\
KC3&WC &    1&   0&    0\\
KC3& CC&    1&   0&    0\\
&&&&\\
MW1&WC &    1&   0&    0\\
MW1& CC&    1&   0&    0\\
&&&&\\
MC2&WC &    1&   0&    0\\
MC2& CC&    1&   0&    0\\
\end{tabular}
\caption{Mann-Whitney results for PD and PF. Rankings are the same for both.} 
\label{fig:an1MW}
\end{figure}

The statistical analysis of \fig{an1MW} shows the win, loss, ties seen in different data sets where the training
set either comes from local data (the WC results) or data from other sites (the CC results).
This figure is sorted by wins-losses seen in a Mann-Whitney U-test (95\%):
\bi
\item For six data sets (\{CM1,KC2,KC3,MC2,MW1,PC1\}.)  within-company data had higher probability of detection and lower probability of false alarm. 
\item Only one set, \{KC1\}, yielded better cross-company results.  
\ei
With results that are statistically superior for six of the seven sets, it would be easy to
completely dismiss the validity of cross-company data. Several of these sets are several places apart. 
Data set KC2 performed particularly well when trained on within-company data. Once the training set was
switched to cross-company, it dropped below sets KC1 and KC3 on probability of detection performance. 

The quartile charts in \fig{an1PD} and \fig{an1PF} reveal a slightly different view of the results.
While the result gap is large enough to stand out on a statistical test, the median probability of false alarm values 
for CC-trained predictors is only 13\% higher than the result for WC data. Probability of detection
results are even closer, WC-trained results are only about 7\% better than those for probability of detection.  

\begin{figure}[!h]
\scriptsize
\begin{center}
\begin{tabular}{c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}c}
\multicolumn{1}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{2-6}
&min& & median & &max\\\cline{2-6}
treatment&0& 25\%& 50\% & 75\%&100\%\\\cline{1-6}
CC & 0.0& 47.1& 63.4& 86.8&100.0& \boxplot{0.0}{47.1}{63.4}{86.8}{13.2}\\ 
WC&0.0& 55.4& 68.1& 82.8&100.0&  \boxplot{0.0}{55.4}{68.1}{82.8}{17.2}  \\
\end{tabular}
\end{center}
\caption{Probability of Detection (PD) results for Naive Bayes on within vs cross-company data. 
Higher values are prefered.
}\label{fig:an1PD}
\end{figure}

\begin{figure}[!h]
\scriptsize
\begin{center}
\begin{tabular}{c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}c}
\multicolumn{1}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{2-6}
&min& & median & &max\\\cline{2-6}
treatment&0& 25\%& 50\% & 75\%&100\%\\\cline{1-6}
CC & 0.0& 12.5& 36.4& 52.4&100.0& \boxplot{0.0}{12.5}{36.4}{52.4}{47.6}\\ 
WC&0.0& 16.7& 31.8& 44.4&100.0&  \boxplot{0.0}{16.7}{31.8}{44.4}{55.6}  \\
\end{tabular}
\end{center}
\caption{Probability of False Alarm (PF) results for Naive Bayes on within vs cross-company data. 
Lower values are prefered.
}\label{fig:an1PF}
\end{figure}

What does this mean for companies building defect predictors?
Unsurprisingly, local data is better.  Within-company data will
naturally reflect the exact software practices of that company.
Logically, cross-company data will be tuned to the environment of
the company that it came from. However, the fact that the median
values were so close is encouraging. The median probability of detection \& false alarm results of
\{68,36\} are well-within established industry bounds~\cite{me07a}.  These initial
results indicate that firms lacking their own data repository can
use cross-company data to build there own local defect predictors. But, as we shall see below,
we should not use imported data without first imposing a {\em relevancy filter}.
