\section{Experiment Replication: Can our results be generalized?}
\label{sec:exp4}
Experiments \# 1 to \# 3 were based on NASA projects.
In order to search evidence for the external validity of the conclusions of those experiments,
we replicate the same experiments on SOFTLAB projects of \tbl{data1}. 

For each SOFTLAB project, we follow the same procedure as in Experiments \# 1 and \#2; i.e.
10\% of the rows of each data set are selected at random for constructing test sets and then defect
predictors are learned from:
\bi
\item CC data: all rows from 7 NASA tables.
\item WC data:  random 90\% rows of remaining SOFTLAB tables\footnote{In order to reflect the use in practice, we do not use the remaining 90\% of the same project for training, we rather use a random 90\% of data from other projects. Please note that all WC experiments in this paper reflects within-company, not within project simulations. Since SOFTLAB data are collected from a single company, learning a predictor on some projects and to test it on a different one does not violate within company simulation.};
\item NN filtered CC data: {\em similar} rows from 7 NASA tables.
\ei 

The SOFTLAB projects include 29 static code features, 17 of which are
 common with the NASA projects. In order to simplify the comparison between these new
projects and Experiment \#1 and \#2, 
 we used only these shared attributes in our CC experiments. 
On the other hand we use all available features in WC experiments for SOFTLAB projects. 
In the following external validity experiment, we treated each NASA project as 
cross- company data for SOFTLAB projects. 

\begin{figure}[!t]
\begin{center}
\begin{tabular}{c}
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Overall results on SOFTLAB data}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & CC&       88&  88&  95&  100& 100& \boxplot{88}{0}{95}{100}{0}\\
    & WC&       35&  40&  88&   100& 100& \boxplot{35}{5}{88}{100}{0}\\\hline
 pf & CC&       52&  59&  65&   68& 68& \boxplot{52}{7}{65}{68}{0}\\
    & WC&        3&  5&  29&   40&  42& \boxplot{3}{2}{29}{40}{2}\\
\hline\end{tabular}
}\\

{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($ar3$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & CC&       88&  88&  88&  88& 88& \boxplot{88}{0}{88}{88}{0}\\
    & WC&       88&  88&  88&   88& 88& \boxplot{88}{0}{88}{88}{0}\\\hline
 pf & CC&       62&  65&  65&   65& 65& \boxplot{62}{3}{65}{65}{0}\\
    & WC&        40&  40&  40&   40&  42& \boxplot{40}{0}{40}{40}{2}\\
\hline\end{tabular}
}\\

{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($ar4$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & CC&       95&  95&  95&  95& 95& \boxplot{95}{0}{95}{95}{0}\\
    & WC&       35&  40&  40&   40& 40& \boxplot{35}{5}{40}{40}{0}\\\hline
 pf & CC&       52&  55&  56&   59& 60& \boxplot{52}{3}{56}{59}{1}\\
    & WC&        3&  3&  3&   5&  5& \boxplot{3}{0}{3}{5}{0}\\
\hline\end{tabular}
}\\

{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Table ($ar5$)}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & CC&       100&  100&  100&  100& 100& \boxplot{100}{0}{100}{100}{0}\\
    & WC&       88&  100&  100&   100& 100& \boxplot{88}{12}{100}{100}{0}\\\hline
 pf & CC&       57&  68&  68&   68& 68& \boxplot{57}{11}{68}{68}{0}\\
    & WC&        29&  29&  29&   29&  29& \boxplot{29}{0}{29}{29}{0}\\
\hline\end{tabular}
}\\
\end{tabular}
\end{center}
\caption{Experiment \#1 results for the SOFTLAB projects. Overall and individual results are shown respectively.}\label{fig:ccwcturkey}
\end{figure}

\begin{figure}
\begin{center}
\renewcommand{\baselinestretch}{0.5}
{\scriptsize\begin{tabular}{l@{~}l@{~}l@{~}r@{~}r@{~}r@{~}r@{~}r@{~}c}
& & &\multicolumn{5}{c}{quartiles}  &\\\cline{4-8}
& rank & & 0 & 25 & 50 & 75 & 100 &\\\hline

AR4&1&   CC&35&90&100&100&100&\boxplot{  35.0}{55}{ 100.0}{ 100.0}{0}\\
&2& NN&65&65&70&70&70&\boxplot{  65.0}{0}{  70.0}{  70.0}{0}\\
&3&   WC&35&40&40&40&45&\boxplot{  35.0}{5}{  40.0}{  40.0}{5}\\\hline

AR3&1&   CC&75&88&88&88&88&\boxplot{  75.0}{13}{  88.0}{  88.0}{0}\\
&1& NN&88&88&88&88&88&\boxplot{  88.0}{0}{  88.0}{  88.0}{0}\\
&1&   WC&88&88&88&88&88&\boxplot{  88.0}{0}{  88.0}{  88.0}{0}\\\hline

AR5&1&   CC&88&100&100&100&100&\boxplot{  88.0}{12}{ 100.0}{ 100.0}{0}\\
&1& NN&100&100&100&100&100&\boxplot{ 100.0}{0}{ 100.0}{ 100.0}{0}\\
&1&   WC&88&100&100&100&100&\boxplot{  88.0}{12}{ 100.0}{ 100.0}{0}\\\hline
\multicolumn{8}{c}{~}&50\%
\end{tabular}}
\end{center}
\caption{
 Experiment \#2 PD results for the SOFTLAB projects where
$NN_{pd} \ge WC_{pd}$. 
Rankings computed via Mann-Whitney (95\% 
confidence) comparing each row to all other rows.}\label{fig:arpd1}
\end{figure}



\begin{figure}[!t]
\begin{center}
\renewcommand{\baselinestretch}{0.5}
{\scriptsize\begin{tabular}{l@{~}l@{~}l@{~}r@{~}r@{~}r@{~}r@{~}r@{~}c}

& & &\multicolumn{5}{c}{quartiles}  &\\\cline{4-8}
& rank & & 0 & 25 & 50 & 75 & 100 &\\\hline

AR3&1& NN&38&38&38&38&40&\boxplot{  38.0}{0}{  38.0}{  38.0}{2}\\
&2&   WC&40&40&40&40&42&\boxplot{  40.0}{0}{  40.0}{  40.0}{2}\\
&3&   CC&38&55&56&60&80&\boxplot{  38.0}{17}{  56.0}{  60.0}{20}\\\hline

AR5&1& NN&21&25&25&25&32&\boxplot{  21.0}{4}{  25.0}{  25.0}{7}\\
&2&   WC&29&29&29&29&29&\boxplot{  29.0}{0}{  29.0}{  29.0}{0}\\
&3&   CC&29&46&50&50&79&\boxplot{  29.0}{17}{  50.0}{  50.0}{29}\\\hline
\multicolumn{8}{c}{~}&50\%
\end{tabular}}
\end{center}
\caption{Experiments \#2 PF results for the SOFTLAB projects
where $NN_{pf} \le WC_{pf}$.}\label{fig:arpf1}
\end{figure}

\begin{figure}
\begin{center}
\renewcommand{\baselinestretch}{0.5}
{\scriptsize\begin{tabular}{l@{~}l@{~}l@{~}r@{~}r@{~}r@{~}r@{~}r@{~}c}
& & &\multicolumn{5}{c}{quartiles}  &\\\cline{4-8}
& rank & & 0 & 25 & 50 & 75 & 100 &\\\hline

AR4&1&   WC&3&3&3&5&7&\boxplot{   3.0}{0}{   3.0}{   5.0}{2}\\
&2& NN&24&25&25&25&28&\boxplot{  24.0}{1}{  25.0}{  25.0}{3}\\
&3&   CC&6&47&62&67&78&\boxplot{   6.0}{41}{  62.0}{  67.0}{11}\\\hline
\multicolumn{8}{c}{~}&50\%
\end{tabular}}
\end{center}
\caption{  Experiment \#2 PF results for the SOFTLAB projects where $NN_{pf} > WC_{pf}$.  }\label{fig:arpf2}
\end{figure}


Figure \ref{fig:ccwcturkey} shows the results:
\bi
\item
The {\em pd} values for CC data increase compared to WC data
with the cost of increased {\em pf}. 
\item
CC data shifts \{Q1,
median\} of {\em pf} from \{5, 29\} to \{59, 65\}.  
\item
For CC data:
\bi
\item 25\% of the $pd$ values are at 100\%.
\item 50\% of the $pd$ values are above 95\%
\item And all the $pd$ values are at or over 88\%.
\ei
\ei

These results also provide evidence for the validity of our conclusions for Experiment \# 3.
In Experiment \# 3, we conclude that the minimum number of instances for training
a defect predictor is around $100-200$ data samples. Please note that SOFTLAB projects
{\em ar3, ar4} and {\em ar5} have \{63,107, 36\} modules respectively,
with a total of 206 modules. Thus, WC results in Figure
\ref{fig:ccwcturkey} are achieved using a minimum of $(63+36) *
0.90 = 90$ data samples (i.e. learn a predictor on {\em (ar3 $+$ ar5)} and test it on {\em ar4}) and a maximum of $(63+107) *
0.90 = 153$ data samples (i.e. learn a predictor on {\em (ar3 $+$ ar4)} and test it on {\em ar5}).

\fig{arpd1} to \fig{arpf2} shows {\em Observation1} (i.e. $NN_{pd} \ge WC_{pd} $) and {\em Observation2} (i.e. $NN_{pf} \le WC_{pf} $) for SOFTLAB projects. Please recall that these observations were mutually exclusive for NASA data. The pattern is similar in SOFTLAB projects:
\bi
\item for {\em ar4} mutual exclusiveness hold: $NN_{pd} \ge WC_{pd}$ and $NN_{pf} > WC_{pf}$
\item  for {\em ar3} and {\em ar5}: $NN_{pf} \le WC_{pf}$. If the observations were mutually exclusive, we would expect  $NN_{pd} < WC_{pd}$. Yet, for {\em pd} $NN_{pd} \ge WC_{pd}$, however this inequality holds with the equity (see \fig{arpd1}) and $NN_{pd} \not > WC_{pd}$
\ei

In summary, 
   the WC, CC and NN patterns found in American NASA rocket software
   are also observed in software controllers of Turkish domestic appliances. While this is not 
   the definitive proof of the external validity of our
   results, we find it a very compelling result that is reproducable in different companies.
