\section{Cross-Company Defect Prediction using Relevancy Filtering}

OURMINE was used to reproduce Turhan et al.'s experiment - with a
Naive Bayes classifier in conjunction with a k-Nearest Neighbor
(k-NN) relevancy filter. Relevancy filtering is used to group similar
instances together in order to obtain a learning set that is
homogeneous with the testing set. Thus, by using a training set
that shares similar characteristics with the testing set, it is
assumed that a bias in the model will be introduced. The k-NN filter
works as follows.  For each instance in the test set: \bi \item the
k nearest neighbors in the training set are chosen.  \item duplicates
are removed \item the remaining instances are used as the new
training set.  \ei

\subsubsection{Building the Experiment}

The entire script used to conduct this experiment is shown in \fig{promiseExp}. 
\begin{figure}
{\scriptsize\begin{verbatim}
1 promiseDefectFilterExp(){
2  local learners="nb"
3  local datanames="CM1 KC1 KC2 KC3 MC2 MW1 PC1"
4  local bins=10
5  local runs=10
6  local out=$Save/defects.csv
7  for((run=1;run<=$runs;run++)); do
8    for dat in $datanames; do
9	       combined=$Data/promise/combined_$dat.arff
10	       shared=$Data/promise/shared_$dat.arff
11	       blabln "data=$dat run=$run" 
12	       for((bin=1;bin<=$bins;bin++)); do
              rm -rf test.lisp test.arff train.lisp train.arff
13	           cat $shared | 
14	           logArff 0.0001 "1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19" > logged.arff
15	           makeTrainAndTest logged.arff $bins $bin
16	           goals=`cat $shared | getClasses --brief`

17	           for learner in $learners; do          
18		            blabln "WC"
19		            $learner train_shared.arff test_shared.arff | gotwant > produced.dat
20		            for goal in $goals; do
21		                extractGoals goal "$dat,$run,$bin,WC,$learner,$goal" `pwd`/produced.dat
24		            done
25		            blabln "WCkNN"
26		            rm -rf knn.arff
27		            $Clusterers -knn 10 test_shared.arff train_shared.arff knn.arff
28		            $learner knn.arff test_shared.arff | gotwant > produced.dat
29		            for goal in $goals; do
30		               extractGoals goal "$dat,$run,$bin,WCkNN,$learner,$goal" `pwd`/produced.dat                          
31             done
32		            blabln "CC"
33		            makeTrainCombined $combined > com.arff	
34		            cat com.arff | logArff 0.0001 "1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19" > logged.arff
35		            $learner logged.arff test_shared.arff | gotwant > produced.dat
36		            for goal in $goals; do
37		               extractGoals goal "$dat,$run,$bin,CC,$learner,$goal" `pwd`/produced.dat
38             done
39             blabln "CkNN"             
40             makeTrainCombined $combined > com.arff  
41             cat com.arff |
42             logArff 0.0001 "1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19" > logged.arff
43             $Clusterers -knn 10 test_shared.arff logged.arff knn.arff
44             $learner knn.arff test_shared.arff | gotwant > produced.dat
45             for goal in $goals; do
46                extractGoals goal "$dat,$run,$bin,CkNN,$learner,$goal" `pwd`/produced.dat
47             done
48    done
49  done
50  done ) | malign | sort -t, -r -n -k 12,12 > $out
\end{verbatim}}
\caption{The OURMINE script used in conducting the WC vs. CC experiment.}\label{fig:promiseExp}
\end{figure}


To begin, the data in this study were the same as used by Gay et
al.~\cite{gay09}; seven PROMISE defect data sets (CM1, KC1, KC2,
KC3, MC2, MW1, PC1) were used to build seven combined data sets
each containing $\frac{6}{7}$-th of the data. For instance, the
file {\em combined\_PC1.arff} contains all seven data sets {\em
except} PC1. This is used as the training set for the cross-company
(CC) data. For example, if we wished to learn on all training data
except for PC1, this would be a valid data set representation for
a cross-company experiment.

Next, as can be seen in line 15 of \fig{promiseExp}, a 10-way cross validation was conducted by calling $makeTrainAndTest$, which is a built-in OURMINE function that randomly shuffles the data and constructs both a test set, containing 10\% of the data, and a training set containing 90\% of the data. This was repeated ten times, and the resulting data was used in proceeding studies. For instance, in lines 18-24, the original test and training sets are used for the first WC study. However, in the WC experiment using relevancy filtering (lines 25-31), the same test set is used, but with the newly created training set. Lines 32-38 show the first CC study. This study is identical to the WC study except that as we saw before, we use $combined\_X.arff$ files, instead of $shared\_X.arff$.

I chose to use a Naive Bayes classifier for this study because this is what was chosen in the original experiment conducted by Turhan et al. in ~\cite{turhan08}, as well as because Naive Bayes has been shown to be competitive on PROMISE defect data against other learners ~\cite{lessmann08}.   


\subsubsection{Results}

Our results for this experiment can be found in \fig{pds} and \fig{pfs}. \fig{pds} shows $pd$ (probability of detection) values sorted in decreasing order, while \fig{pfs} shows $pf$ (probability of false alarm) values sorted in increasing order. These values are calculated as follows.
If $\{a,b,c,d\}$ are the
true negatives, 
false negatives, 
false positives, and 
true positives
(respectively) found by a binary detector then 
$pd=recall=\frac{d}{b+d}$ and
$pf=\frac{c}{a+c}$.
Note that a higher $pd$ is better, while a lower $pf$ is better. 


The last column of each figure shows quartile charts of the methods' $pd$ and $pf$ values. The black dot in the center of each plot represents the median value, and the line going from left to right from this dot show the second and third quartile respectively.

Column one of each figure gives a method its rank based on a Mann-Whitney test at 95\% confidence. A rank is determined by how many times a learner or learner/filter wins compared to another. The method that wins the most number of times is given the highest rank. 

%\begin{figure}
%\small
%\begin{center}
%\begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
%\multicolumn{2}{c}{~}&\multicolumn{3}{c}{pd}&2nd quartile\\
%\multicolumn{2}{c}{~}&\multicolumn{3}{c}{percentiles}&median,\\\cline{3-5}
%Rank&Treatment& 25\%& 50\% & 75\%&3rd quartile\\\hline
%1&WC (local data) + relevancy filter& 66& 73& 80&  \boxplot{0}{66}{73}{80}{3}  \\
%2&CC (imported data) + relevancy filter& 57& 71& 83&  \boxplot{0}{57}{71}{83}{5}  \\
%2&WC (local data)  & 59& 69& 83&  \boxplot{0}{59}{69}{83}{7}  \\
%3&CC (imported data)  & 49& 66& 87&  \boxplot{0}{49}{66}{87}{7}   \\\hline
%\multicolumn{5}{c}{~}&~~~~~0~~~~~~~~50~~~~100
%\end{tabular}
%\end{center}
%\caption{Probability of Detection (PD) results, sorted by median values.  }\label{fig:pds}
%\end{figure}
%\begin{figure}
%\small
%\begin{center}
%\begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
%\multicolumn{2}{c}{~}&\multicolumn{3}{c}{pf}&2nd quartile\\
%\multicolumn{2}{c}{~}&\multicolumn{3}{c}{percentiles}&median,\\\cline{3-5}
%Rank&Treatment& 25\%& 50\% & 75\%&3rd quartile\\\hline
%1&CC (imported data) + relevancy filter& 17& 29& 43&  \boxplot{0}{17}{29}{43}{3}  \\
%1&CC (imported data)  & 13& 34& 51&  \boxplot{0}{13}{34}{51}{5} \\
%2&WC (local data)  & 17& 30& 41&  \boxplot{0}{17}{30}{41}{7}  \\
%3&WC (local data) + relevancy filter & 20& 27& 34&  \boxplot{0}{20}{27}{34}{8}  \\\hline
%\multicolumn{5}{c}{~}&~~~~~0~~~~~~~~50~~~~100
%\end{tabular}
%\end{center}
%\caption{Probability of False Alarm (PF) results, sorted by median values.  }\label{fig:pfs} \end{figure}

The following are important conclusions derived from these results:

\begin{itemize}
\item 
When CC data is used, relevancy filtering is crucial. According to our results, cross-company data with no 
filtering yields the worst $pd$ and $pf$ values.
\item{When relevancy filtering is performed on this CC data, we obtain better $pd$ and $pf$ results than using just CC and Naive Bayes.}
\item{When considering only filtered data or only unfiltered data, the highest $pd$ and lowest $pf$ values are obtained by using WC data as opposed to CC data. This suggests that WC data gives the best results.}
\end{itemize}

These finds were consistent with Turhan et al.'s results: 

\begin{itemize}
\item{Significantly better defect predictors are produced from using WC data.}
\item{However, CC data leads to defect predictors nearly as effective as WC data when using relevancy filtering.}
\end{itemize}

Thus, this study also makes the same conclusions as Turhan et al. A company should use local data to develop defect predictors if that local development data is available. However, if local data is not available, relevancy-filtered cross-company data provides a feasible means to build defect predictors.

\begin{figure}
\small
\begin{center}
\begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
\multicolumn{2}{c}{~}&\multicolumn{3}{c}{pd}&2nd quartile\\
\multicolumn{2}{c}{~}&\multicolumn{3}{c}{percentiles}&median,\\\cline{3-5}
Rank&Treatment& 25\%& 50\% & 75\%&3rd quartile\\\hline
1&WC (local data) + relevancy filter& 66& 73& 80&  \boxplot{0}{66}{73}{80}{3}  \\
2&CC (imported data) + relevancy filter& 57& 71& 83&  \boxplot{0}{57}{71}{83}{5}  \\
2&WC (local data)  & 59& 69& 83&  \boxplot{0}{59}{69}{83}{7}  \\
3&CC (imported data)  & 49& 66& 87&  \boxplot{0}{49}{66}{87}{7}   \\\hline
\multicolumn{5}{c}{~}&~~~~~0~~~~~~~~50~~~~100
\end{tabular}
\end{center}
\caption{Experiment \#2 (WC vs. CC). Probability of Detection (PD) results, sorted by rank then median values.  }\label{fig:pds}
\end{figure}
\begin{figure}
\small
\begin{center}
\begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
\multicolumn{2}{c}{~}&\multicolumn{3}{c}{pf}&2nd quartile\\
\multicolumn{2}{c}{~}&\multicolumn{3}{c}{percentiles}&median,\\\cline{3-5}
Rank&Treatment& 25\%& 50\% & 75\%&3rd quartile\\\hline
1&CC (imported data) + relevancy filter& 17& 29& 43&  \boxplot{0}{17}{29}{43}{3}  \\
1&CC (imported data)  & 13& 34& 51&  \boxplot{0}{13}{34}{51}{5} \\
2&WC (local data)  & 17& 30& 41&  \boxplot{0}{17}{30}{41}{7}  \\
3&WC (local data) + relevancy filter & 20& 27& 34&  \boxplot{0}{20}{27}{34}{8}  \\\hline
\multicolumn{5}{c}{~}&~~~~~0~~~~~~~~50~~~~100
\end{tabular}
\end{center}
\caption{Experiment \#2 (WC vs. CC). Probability of False Alarm (PF) results, sorted by rank then median values.  }\label{fig:pfs} \end{figure}
