
%
%  $Description: Author guidelines and sample document in LaTeX 2.09$ 
%
%  $Author: ienne $
%  $Date: 1995/09/15 15:20:59 $
%  $Revision: 1.4 $
%

\documentclass[times, 10pt,twocolumn]{article} 
\usepackage{cite,latex8}
\usepackage{times,url}
\usepackage{graphicx}
\newcommand{\boxplot}[5]{
	\begin{picture}(100,10)
		%\put(0,0){\line(0,1){8}}
		%\put(100,0){\line(0,1){8}}
		\put(#1,4){\line(1,0){#2}}
		\put(#3,4){\circle*{6}}
		\put(#4,4){\line(1,0){#5}}
		\put(50,0){\line(0,1){8}}
	\end{picture}
}
\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{itemize}
   \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{itemize}}
\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}

\newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}
 \newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
 \newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}

%\documentstyle[times,art10,twocolumn,latex8]{article}

%------------------------------------------------------------------------- 
% take the % away on next line to produce the final camera-ready version 
\pagestyle{empty}

%------------------------------------------------------------------------- 
\begin{document}

\title{Regularities in Learning Defect Predictors}

\author{Burak Turhan, Ayse Bener\thanks{The research described in this paper was supported by Bo{\u g}azi{\c c}i  University 
research fund under grant number BAP-06HA104 and at West Virginia 
University under grants with NASAs Software Assurance Research Program. 
Reference herein to any specific commercial product, process, or service 
by trademark, manufacturer, or otherwise, does not constitute or imply its 
endorsement by the United States Government.}\\
Dept. of Computer Engineering \\ Bogazici University\\ 
34342 Bebek, Istanbul, Turkey\\ \url{{turhanb , bener}@boun.edu.tr}\\
\and
Tim Menzies\\
Lane Dept. of CS\&EE \\West Virginia University \\ 
Morgantown, WV, USA\\\url{tim@menzies.us}\\}

\maketitle
\thispagestyle{empty}


\input{abstract}
\input{introduction}
\input{related}
%------------------------------------------------------------------------- 
%------------------------------------------------------------------------- 

%------------------------------------------------------------------------- 

In prior work~\cite{me08d,me08e}, we have used the NASA and SOFTLAB data of \fig{data}
to explore learning defect predcitors using data miners. 
To learn defect predictors we use a Naive Bayes data miner since prior work~\cite{me07b} could not
find a better data miner for learning defect predictors.
In all our experiments, the data was pre-processed as follows:
\bi
\item
Since the number of features in each data table is not consistent,
we restricted our data to only the features shared by all data sets.
\item
Previously~\cite{me07b}, 
we have observed that
all the numeric distributions in the \fig{data}
data are exponential in nature. It is therefore useful to apply a  ``log-filter'' to all numerics $N$ with $log(N)$.
This spreads out exponential
curves more evenly across the space from the minimum to maximum values
(to avoid numerical errors with  $ln(0)$,
all numbers under 0.000001 are replaced with  $ln(0.000001)$). 
\ei
\begin{figure}
\scriptsize
\begin{center}
\begin{tabular}{ll|rrr@{.}l}
     &         & (\# modules)         &          &  &\\
source & data &  examples & features  & \multicolumn{2}{c}{\%defective}\\\hline
%pc5  & C++ & 17,186   & 22          & 3&0\\
%mc1  & C++ & 9,466    & 22          & 0&71\\
%pc2  & C++ & 5,589    & 22         & 0&41 \\
%pc3  & C++ & 1,563    & 22          & 10&23\\
%pc4  & C & 1,458    & 22          & 12&2\\
NASA & pc1   & 1,109    & 22          & 6&94\\
NASA & kc1  & 845      & 22          & 15&45\\
NASA & kc2  & 522      & 22          & 20&49\\
NASA & cm1   & 498      & 22          & 9&83\\
NASA & kc3  &  458      & 22          & 9&38\\
NASA & mw1  &  403      & 22          & 7&69\\
NASA & mc2  & 61      & 22          & 32&29\\
SOFTLAB & ar3 & 63 & 29 & 12&70\\
SOFTLAB & ar4 &  107 & 29 & 18&69\\
SOFTLAB & ar5 &  36 & 29 & 22&22\\
OPEN SOURCE & cav90 &  1184 &   26 & 0&40\\
OPEN SOURCE & cav91 &  1169 &   26 & 0&20\\
OPEN SOURCE & gcc &  116 & 26 & 100&0 \\ \\

\end{tabular}
\end{center}
\caption{Datasets used in this study. All the NASA and SOFTLAB data sets are available at \protect\url{http://promisedata.org}.}\label{fig:data}
\end{figure} 

Inspired by the recent systematic review of within vs cross company effort estimation studies by Kitchenham et al.~\cite{kitch07}, we have done extensive experiments on Promise data tables to analyze predictor behavior using a) local data (within the same company) b) imported data (cross company). For each NASA and SOFTLAB table of \fig{data}, we built test sets from
10\% of the rows, selected at random.  Then we learned defect
predictors from:
\bi
\item the other 90\% rows of the corresponding table (i.e. local data).
\item 90\% rows of the other tables combined (i.e. imported data).
\ei

We repeated this procedure 100 times, each time randomizing the order of the rows in each table, in order to control  {\em order effects} (where the 
learned theory is unduly affected by the order of the examples). 
We measured the performance of predictor using $pd$, $pf$ and $balance$.
If $\{A,B,C,D\}$ are the
true negatives,
false negatives, 
false positives, and 
true positives
(respectively) found by a defect predictor, then:

{\footnotesize
\begin{eqnarray}
pd=&recall=&D/(B+D)\\
pf=&&C/(A+C)\\\label{eq:bal}
bal=&balance=&1 - \frac{\sqrt{\left(0 - pf\right)^2 + \left(1 - pd\right)^2}}{\sqrt{2}}
\end{eqnarray}
}
All these values range zero to one.
Better and {\em larger} balances fall  {\em closer} to the desired  zone 
of no false alarms ($pf=0$) and 100\% detection ($pd=1$). We then used the Mann-Whitney U test~\cite{mann47} in order to test
for statistical difference.

Our results are visualized in \fig{promise} as {\em quartile chaorts}.
Quartile charts are generated from sorted sets of results, divided into four subsets of (approx) same cardinality.
For example these numbers have four quartiles:
\[
{\footnotesize
\{\overbrace{4,7,15}^{q1},20,31,\overbrace{40}^{median},52, 64, \overbrace{70,81,90}^{q4}\}}
\]
These quartiles can be drawn as follows:
the upper and lower quartiles are  marked with black lines;
the
median is marked with a black dot; and
vertical bars  are added to
mark the 50\% percentile value.
For example, the above numbers can be drawn as:
{\small
\[0\%\; \boxplot{4}{11}{40}{70}{20}\; 100\%\]}

In a finding consistent with our general thesis (that software artifacts conform to very regular
statistical patterns),
\fig{promise} shows the same stable and useful regularity
occuring in both
seven NASA data sets and
three
SOFTLAB data sets:

\bi
\item Using imported data dramatically increased the probability of detecting defective modules (for NASA: 74\% to 94\% median $pd$;
for SOFTLAB: 88\% to 95\% median $pd$);
\item But imported data also dramatically increased the false alarm rate
(for NASA: 29\% to 68\% median $pf$;
for SOFTLAB: 29\% to 65\% $pd$) . 
Our suspicion is that the reason for the high false alarm rates was the irrelevancies introduced by imported data.
\ei
\begin{figure}[!t]
\begin{center}
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Nasa on local Nasa data.}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd  & Local&       0&  60&  75&   82& 100& \boxplot{0}{60}{75}{82}{18}\\
 pf  & Local&        0&  24&  29&   35&  73& \boxplot{0}{24}{29}{35}{38}\\
\hline\end{tabular}

~\\~\\

\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Nasa on imported Nasa data.}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & Imp.&       81&  84&  94&  98& 99& \boxplot{81}{3}{94}{98}{1}\\
 pf & Imp.&        26&  60&  68&   90& 94& \boxplot{26}{23}{68}{90}{4}\\ \hline\end{tabular}

~\\~\\

%\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
%\multicolumn{7}{l}{Nasa on nn-imported Nasa data.}\\\hline
%&treatment & min & Q1 & median & Q3 & max  \\\hline
% pd & NN Imp.&       50&  59&  68&  73& 79& \boxplot{50}{9}{68}{73}{6}\\
% pf & NN Imp.&        15&  20&  27&   38& 41& \boxplot{15}{5}{27}{38}{3}\\
%\hline\end{tabular}

%~\\~\\

\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Softlab on local Softlab data.}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & Local&      35&  40&  88&   100& 100& \boxplot{35}{5}{88}{100}{0}\\
 pf & Local&        3&  5&  29&   40&  42& \boxplot{3}{2}{29}{40}{2}\\
\hline\end{tabular}

~\\~\\


\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{Softlab on imported Nasa data.}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & Imp.&      88&  88&  95&  100& 100& \boxplot{88}{0}{95}{100}{0}\\
  pf & Imp.&        52&  59&  65&   68& 68& \boxplot{52}{7}{65}{68}{0}\\ \hline\end{tabular}

%~\\~\\

%\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
%\multicolumn{7}{l}{Softlab on nn-imported Nasa data.}\\\hline
%&treatment & min & Q1 & median & Q3 & max  \\\hline
% pd & NN Imp.&      70&  70&  88&  88& 88& \boxplot{70}{0}{88}{88}{0}\\
%  pf & NN Imp.&        24&  28&  32&   38& 44& \boxplot{24}{4}{32}{38}{6}\\
%
%\hline\end{tabular}

}
\end{center}
\caption{Quartile charts for NASA and SOFTLAB data.
Numeric results on left; quartile charts on right. ``Q1'' and ``Q3'' denote
the 25\% and 75\% percentile points (respectively). 
}\label{fig:promise}
\end{figure}

We have then designed a set of experiments for NASA data tables to see the effects of sampling strategies on the performance and to determine the lower-limit on the number of samples for learning defect predictors, i.e. the point  where a plateau is observed in the performance measure. In those experiments we applied over/under and micro-sampling (see \fig{micro}) to the data tables and observed that:

\bi
\item the performance of predictors does not improve with over sampling.
\item under-sampling improves the performance of certain predictors, i.e. decision trees, but not of Naive Bayes~\cite{me08d,me08e}.
\item with micro-sampling, the performance of predictors stabilize after a mere number of defective and defect-free examples, i.e. 50 to 100 samples.
\ei
\begin{figure}[!t]
\begin{center}
\small
\begin{tabular}{|p{.95\linewidth}|}\hline
{\em Micro-sampling} is a generalization of under-sampling such that given $N$ defective modules
in a data set, \[M \in \{25,50,75,.\} \le N\] 
defective modules were selected
at random. Another $M$ non-defective modules are selected, at random. The combined $2M$ data set is
then passed to a 10*10-way cross validation. 

Formally, under-sampling is a micro-sampling where $M=N$. Micro-sampling explores
training sets of size up to N, standard under-sampling just explores once data set of
size $2N$.

\fig{under} shows the results of an micro-sampling study where $M\in\{25,50,75,...\}$
defective modules were selected at random, along with an equal $M$ number of defect-free modules.
Note the visual pattern: increasing data does not necessarily improve $balance$.

Mann-Whitney tests were applied to test this visual pattern. 
Detectors learned from small $M$ instances do as well
as detectors learned from any other number of instances.
\bi
\item
For six data sets,
\{CM1,KC2, KC3, MC2, MW1, PC1\},
micro-sampling at $M=25$ did just as well as anything larger sample size.
\item
For one data set \{KC1\} best results were seen at $M=\{575\}$. 
However, in all by one case
$M=25$ did as well as any larger value.
\ei

\\\hline
\end{tabular}
\end{center}
\caption{Micro-sampling.}\label{fig:micro}
\end{figure}
\begin{figure*}[!t]
\begin{center}
\begin{tabular}{ccc}
\includegraphics[angle=90,width=2.1in]{underCM1.pdf}&
\includegraphics[angle=90,width=2.1in]{underKC1.pdf}&
\includegraphics[angle=90,width=2.1in]{underKC2.pdf}\\
\includegraphics[angle=90,width=2.1in]{underKC3.pdf}&
\includegraphics[angle=90,width=2.1in]{underMC2.pdf}&
\includegraphics[angle=90,width=2.1in]{underMW1.pdf}\\
\includegraphics[angle=90,width=2.1in]{underPC1.pdf}&
\end{tabular}
\end{center}
\caption{Micro-sampling results for NASA data}\label{fig:under}
\end{figure*}
The last observation suggests that  
the number of cases that  must be reviewed in order to arrive at the performance ceiling of a defect 
predictor is very small:  as low as 50 randomly
selected modules (25 defective
and 25 non-defective). In \fig{under} for Nasa tables, we visualize number of training examples (increments of 25) vs. balance performance of Naive Bayes predictor. It is clear that the performance does not improve with more training examples, indeed it may deteriorate with more examples.

We now report the results of the same experiment for SOFTLAB data. Note that SOFTLAB tables {\em ar3, ar4} and {\em ar5} have \{36,63,107\} modules respectively,
with a total of 206 modules of which only 36 are defective. Thus, local results in \fig{promise} are achieved using a minimum of $(36+63) *
0.90 = 90$ and a maximum of $(107+63) *
0.90 = 153$ examples. Nevertheless we repeat the micro-sampling experiment for SOFTLAB data tables, with increments of 5 due to relatively lower number of defects. 
In \fig{ar}, we plot the results for SOFTLAB data tables. We observe the same pattern as in \fig{promise}: performance tends to stabilize after a small number of training examples.

\begin{figure*}[!t]
\begin{center}
\begin{tabular}{ccc}
\includegraphics[height=1.25in,width=2.1in]{ar3.pdf}&
\includegraphics[height=1.25in,width=2.1in]{ar4.pdf}&
\includegraphics[height=1.25in,width=2.1in]{ar5.pdf}\\


\end{tabular}
\end{center}
\caption{Micro-sampling results for SOFTLAB data}\label{fig:ar}
\end{figure*}


\section{External Validity}

%Based on \fig{promise}, we concluded previously~\cite{me08e} that even if
%data was available from other sites, {\em it is a better policy to ignore
%it and use local data}.
Results for NASA and SOFTLAB data tables suggest that practical defect predictors can be learned using only a handful of examples. In order to allow for generalization,
it is appropriate to question the external validity
of the above two results:  
The data used for those results come from very different sources:
\bi
\item
The SOFTLAB data were collected from a Turkish white-goods manufacturer
(see the  the datasets (\{ar3, ar4, ar5\}) from
\fig{data})
building 
controller software for a washing machine, a dishwasher and a
refrigerator. 
\item
On the other hand, NASA software are ground and flight control projects for aerospace applications, each developed by different teams at different locations .  
\ei
The development practices from these two organizations are very different:
\bi
\item The SOFTLAB software were built in a profit- and revenue-driven commercial organization, whereas NASA is a cost-driven government entity
\item The SOFTLAB software were
are developed by very small teams (2-3 people) working in the same
physical location while the NASA
software was built by much larger team spread around the United States.
\item The SOFTLAB development was carried out in an ad-hoc, 
informal way rather than formal, process oriented approach used
at  NASA. 
\ei

The fact that the same defect detection
patterns hold  for such radically different kinds of organization
is a strong argument for the external validity of our results.
However, an even stronger argument would be that the patterns
we first saw at NASA/SOFTLAB are also found in software developed
at other sites. The rest of this paper collects evidence
for that stronger argument.


%Motivated by the above reasoning, we now present another set of experiments, this time using nearest-neighbor sampling rather than random sampling. We will show that nearest neighbor sampling reduces the conjectured irrelevancies in imported data to some extent. The experimental rig is the same as before, however, we do not use the random 90\% of imported data, rather we use the nearest neighbors of local data among the imported data. Our preliminary analysis shows that this latter approach lies somewhere between using local data and imported data, while closer to former than the latter. 



%In Figure \ref{fig:promise},  the visual pattern is clear: imported data increase probability of detections along with false alarm rates, and nn-imported data remove the high false alarm rates, again, along with probability of detections. This outcome does not contradict our previous conclusion that imported data should only be used in mission critical projects. While nn-imported data decreases false alarm rates, predictors learned from local data still performs the best. Thus, companies should still rely on using local data for learning predictors. 

   
%------------------------------------------------------------------------- 
\Section{Validity in Open Source Domain}

This section checks for the above patterns in two open source 
projects:
\bi
\item
two versions of an anti-virus project: Clam AV v0.90 and v0.91;
\item
a subset of defective modules of the GNU gcc compiler. 
\ei
In \fig{data}, these are denoted as cav90,
cav91
and gcc, respectively.
Note that these are very different projects, build by different developers
with very different purposes. Also note that the development processes
for the open source projects are very different to the NASA and SOFTLAB
projects studied above. Whereas NASA and SOFTLAB were developed
by centrally-controlled top-down 
management teams, cav90, cav91 and gcc were build in a highly distributed manner.
Further, unlike our other software,
gcc has been under extensive usage and maintenance for over
a decade.



\begin{figure}[!t]
\begin{center}
{\scriptsize
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{cav90 on local cav90 data.}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & Local&       0&  40&  40&  60& 100& \boxplot{0}{40}{40}{60}{40}\\
 pf & Local&        22&  32&  35&   37& 44& \boxplot{22}{10}{35}{37}{44}\\
\hline\end{tabular}

~\\~\\
 
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{cav90 on imported Nasa data.}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & Imp.&       49&  49&  51&  51& 53& \boxplot{49}{0}{51}{51}{2}\\
 pf &Imp.&        35&  36&  37&   37& 38& \boxplot{35}{1}{37}{37}{1}\\ \hline
 \end{tabular}

~\\~\\


\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{cav91 on local cav91 data.}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & Local&       0&  67&  67&  100& 100& \boxplot{0}{67}{67}{100}{0}\\
 pf & Local&        17&  25&  28&   30& 39& \boxplot{17}{8}{28}{30}{9}\\
\hline\end{tabular}

~\\~\\
 
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{cav91 on imported Nasa data.}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & Imp.&       77&  77&  77&  77& 77& \boxplot{77}{0}{77}{77}{0}\\
 pf &Imp.&        36&  38&  38&   39& 40& \boxplot{36}{2}{38}{39}{1}\\ \hline
 \end{tabular}

~\\~\\

\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{gcc defects on imported cav91 data.}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & Imp.&       53&  53&  53&  53& 53& \boxplot{53}{0}{53}{53}{0}\\
 %pf &Imp.&        N/A&  N/A&  N/A&   N/A& N/A& %\boxplot{0}{100}{ -1}{0}{0}
 \hline
 \end{tabular}

~\\~\\ 
 
\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
\multicolumn{7}{l}{gcc defects on imported Nasa data.}\\\hline
&treatment & min & Q1 & median & Q3 & max  \\\hline
 pd & Imp.&       60&  60&  60&  60& 60& \boxplot{60}{0}{60}{60}{0}\\
 %pf &Imp.&        N/A&  N/A&  N/A&   N/A& N/A& %\boxplot{0}{100}{ -1}{0}{0}
 \hline
 \end{tabular}

%\begin{tabular}{r@{}r|r@{~}r@{~}r@{~}r@{~}r@{~}|l}
%\multicolumn{7}{l}{Open Source on nn-imported Nasa data.}\\\hline
%&treatment & min & Q1 & median & Q3 & max  \\\hline
%  pd & NN Imp.&       73&  73&  73&  73& 73& \boxplot{73}{0}{73}{73}{0}\\
% pf &NN Imp.&        27&  28&  28&   29& 30& \boxplot{27}{1}{28}{29}{1}\\
%
%\hline\end{tabular}
}
\end{center}
\caption{Quartile charts for OPENSOURCE data.}\label{fig:os}
\end{figure}

Local/ imported data experiments are once more applied on cav90, cav91 and gcc data tables and the results are visualized in Figure \ref{fig:os}. We again used Nasa tables as imported data, since they provide a large basis with 5000+ samples. Note that partial gcc data includes only a sample of defects, thus we are not able to make a 'local data' analysis for gcc. Rather, we report the detection rates of predictors built on imported data from Nasa and cav91.
These predictors can correctly detect up to median $60\%$ of the subset of bugs that we were able to manually match to functional modules.

Recall that cav91 has a defect rate of $0.20\%$. The probability of detection rates for cav91 are median $67\%$ and $77\%$ for local and imported data respectively, which is another evidence on the usefulness of statistical predictors.    

At the first glance, the patterns in \fig{os} seem a bit different than those in \fig{promise}. There are still increases in probability of detection and false alarms rates, though not dramatically. However, this is not a counter example of our claim. We explain this behavior with the following assertions: 

For our experiments:
\bi
\item in commercial software analysis, local data corresponds to single projects developed by a relatively small team of people in the same company and with certain business knowledge.
\item in commercial software analysis, imported data corresponds to a variety of projects developed by various people in different companies and spans a larger business knowledge.  
\item in open source software analysis,  local data corresponds to single projects developed by a larger team of people at different geographical locations with various backgrounds.
%\item In open source software analysis, imported data corresponds to variety of projects developed by various people in different companies.  
\ei 

We argue that, the above assertions differentiate the meaning of local data for open source projects from commercial projects. The nature of open source development allows the definition of local data to be closer to commercial imported data, since both are developed by people at different sites with different background. That's the reason why adding commercial imported data does not add as much detection capability as it does for commercial local data. Furthermore, the false alarms are not that high since there are less irrelevancies in local open source data than commercial imported data, which is the cause of high false alarms. That's because open source local data consist of a single project and commercial imported data consist of several projects, which introduce irrelevancies.


We also repeat the 10*10 way cross-validation micro-sampling experiment for cav90 and cav91 data tables, this time with increments of $5$ due to limited defect data (\fig{under} used increments of $25$). In \fig{opensource} we see a similar pattern: 
\bi
\item Balance values tend to converge using only 20-30 samples of defective and non-defective modules. 
\item Using less examples produce unstable results, since there are not enough samples to learn a theory.
\item For cav90, using more than 60 training examples affects the stability of the results.
\ei

The importance of \fig{opensource} results is that our results in two commercial domains, considering the minimum number of examples to train a predictor, are once more validated in a third, open-source domain.   

\begin{figure}[!t]
\begin{center}
\begin{tabular}{c}
\includegraphics[height=1.25in,width=2.1in]{cav90.pdf}\\
\includegraphics[height=1.25in,width=2.1in]{cav91.pdf}\\


\end{tabular}
\end{center}
\caption{Micro-sampling results for OPEN-SOURCE data}\label{fig:opensource}
\end{figure}



%
%\begin{figure}
%\begin{center}
%\begin{tabular}{ccc}
%\includegraphics[width=3in]{v91gcc.pdf}
%\end{tabular}
%\end{center}
%\caption{We learn predictors using only a random 30\% of cav91 data, including 7 local bug reports on the average (upto vertical line). Then we introduced gcc bugs incrementally to learn more predictors.  
%}\label{fig:v91self+gcc}
%\end{figure}
%------------------------------------------------------------------------- 
\Section{Discussions}

We can summarize our results on commercial and open-source data as follows:

\bi
\item Using imported data not only increases the detection capability of predictors, but also their false alarm rates. This effect is less visible in open source domain for the reasons discussed above.
\item Predictors can be learned using only a handful of data samples. In both commercial and open-source domain the performance of predictors converge after a small number of training examples. Providing further training instances may cause variations in predictor performances.
\item These results are generalizable through different cultural and organizational entities, since same patterns are observed in NASA, SOFTLAB and OPEN-SOURCE data tables.
\ei

Note that these result are observed:
in a variety of projects (i.e. 7 NASA, 3 SOFTLAB, 2 OPENSOURCE) 
from different domains (i.e. commercial and open-source)
spanning a wide range time interval (i.e. from 1980's to 2007). Therefore, we assert that:

 {\em There exists repeated patterns in software that can be discovered and explained by simple models with minimal information no matter what the underlying seemingly random and complex processes or phenomena are .} 




This assertion should be processed carefully. We do not mean that {\em everything} about software can be controlled through patterns. Rather, we argue that these patterns exist and are easy to discover. It is practical and cost effective to use them as guidelines in order to understand the behavior of software and to take corrective actions. In this context we will propose two directions for ASE research in our conclusions.  
%------------------------------------------------------------------------- 
\Section{Conclusion}

Automated analysis methods cannot offer 100\% certification guarantee, however
they can be useful to augment more expensive methods such that:

\bi
\item Defect predictors learned on static code attributes achieve detection rates better than manual inspections.
\item Feeding these predictors with third party data (i.e. imported data) improves their detection rates further.
\item However, this comes at the cost of increase false alarm rates
\item These predictors can be learned with as low as 30 samples.
\ei

Building such models is easy, fast and effective in guiding manual test effort to correct locations.
What is more important is that these automated analysis methods are applicable in different domains. We have previously shown their validity in two commercial  domains and in this paper we observe similar patterns in two projects of the open source domain. We have also shown that although these models are sensitive to the information level in the training data (i.e. local/ imported),  they are not affected by the organizational differences that generate them.

Based on our results, we argue that no matter how complicated the underlying processes may seem, software has a statistically predictable nature. Going one step further, we claim that the patterns in software are not limited to individual projects or domains, rather they are generalizable through different projects and domains.

Therefore, we suggest two directions for ASE research:
\bi
\item
One direction should explore software analysis using rigorous formalisms that offer ironclad
gaureentees of the corrections of a code (e.g. interactive theorem proving, model  checking,
or the 
correctness preserving transoftations discussed by Doug Smith\footnote{Keynote address, ASE'07}).
This approach is required
for the analysis of mission-critical
software that must always function correctly. 
\item 
Another direction should explore automatic methods with a stronger focus on maximizing
the effectiveness of the  analysis while minimizing the associated cost.
\ei
There has been some exploration of the second approach using
lightweight formal methods (e.g.~\cite{easter98}) 
or formal methods that emphasis the usability and ease of use of the 
tool (e.g.~\cite{levenson96,heitmeyer96automated}).
However,  our results also show that there exists an under-explored space of extrememly
cost-effective automated analysis methods.
%------------------------------------------------------------------------- 

\bibliographystyle{latex8}
\bibliography{ase,refs}


\end{document}

