
\section{Activity and Inspection Optimization}

To assess the value of the selected metrics in \fig{metrics}, we distilled them for \NSystems open-source Java projects from the Helix project and applied them to build defect predictors. As shown below, the median value for the $\frac{learning}{oracle}$ ratio will be 96\%, that is, very close to the theoretical upper bound possible for any defect predictor for the the task of inspection optimization.

\begin{figure}[t]
\footnotesize
\begin{center}
\begin{tabular}{|r|l|}\hline
\bf{System} & \bf{Description} \\ \hline
ant	& Build management system \\
ivy & Dependency manager \\
jedit & Text editor \\
lucene & Text search engine \\
poi & API for Office Open XML standards \\
synapse & Enterprise service bus\\
velocity & Template language engine \\
xalan & XSLT processor \\
xerces & XML processor \\ \hline
\end{tabular}
\end{center}
\caption{Java systems used in this study.}\label{fig:projects}
\end{figure}

\subsection{Data}

%\subsection{Input Data Set}
%The field of software engineering has made a number of advances
%over the past few decades. However, only a small amount of serious
%effort has been directed towards creating and making available data
%sets that allow researchers to easily conduct new experiments and
%more importantly replicate other experiments. The Helix data set
%\cite{Helix10a} fills this specific gap along with similar efforts
%by Tempero et. al. \cite{Tempero10a} and the Promise data set
%currently maintained by Menzies et. al \cite{me07c}.

The data used in this studies was built as a join between two complementary data sets:

\bi
\item The PROMISE repository~\cite{me07c} contains defect information for various open-source object-oriented systems. The defect data for this study was collected by Jureczko \cite{Jureczko10a}.

\item The Helix repository~\cite{Helix10a,Vasa10a} provides static source code metrics for a compilation of release histories of non-trivial Java open-source software systems.
\ei

The joined data sets represent \NSystems releases of the projects listed in \fig{projects}. All projects are ``long term" (at least 15 releases span over a development period of 36 months or more) and comprise more than 100 classes each. In addition, every project can be characterized as either \emph{application}, \emph{framework}, or \emph{library}, a broad ``binning" strategy that reflects the inherent, yet recurring, differences in software design and composition. For a detailed description of these data sets, see~\cite{Vasa10a}.

%The rationale for the collected information is given in \fig{metrics}. 

For \LOC (\ie, the \emph{Lines of Codes}) we use an estimator based on the size of the compiled byte code rather than the actual source code. The byte code provides us with a noise-free image of the class's defined functionality. 
\LOC of a class $C$ is given as the sum of the following components extracted from the binaries:

\bi

\item out-degree of $C$ line(s) for {\tt import} statements
\item 1 line for the class declaration
\item 1 line for super class declaration if not {\tt java.lang.Object}
\item 1 line for each interface implemented by $C$
\item 1 line for each field defined in class $C$
\item 1 line for each method $m$ defined in class $C$, plus

\bi

\item \# parameters of $m$
\item \# {\tt throws} defined by $m$
\item {\tt MaxLocals} attribute (\ie, local variables) of $m$
\item \# byte code instructions in $m$

\ei

\ei

%% Tim:
%so what does that mean about the process around the inspection? this
%reads to me like there should be more text on p3 (the dot points right
%hand side) on how these inspections are conducted

%\noindent
We selected these components as they provide a very consistent approximation of the size of source code independent of the actual coding style used. The LOC estimator correlates very well with the lines of source code (cf. \fig{locfit}). 
%
%Furthermore, a key strength of our approach is that byte code processing guarantees that unused code is ignored. 
%
Furthermore, for the purpose of inspection optimization, an added benefit of processing byte code rather than source is that
the data miner will only report those classes  that actually appear in the released version. That is, the secondary human inspection team is given further guidance to focus its QA effort.
%
Previous research \cite{Krinke01a, Antoniol00c, Kontogiannis96a, Johnson94a} found that, in general,
%not all of the source code is part of the final release build. 
not all parts of the code base are included in the final release build. 
This is due to the release build configuration settings used.
%
%This can happen since the actual set of source code 
%that is incorporated into the released version 
%depends on the configuration setting used in the build scripts. 
%
%By resorting to byte code processing, we can guarantee that the code deemed fit for release by developers is the one that is being analyzed. 
%
%Implicit in our approach is the assumption that developers would not actively inspect unused code for defects. 
%
Hence, processing 10\%  of the classes as per byte code, is equivalent to analyzing 10\% of the active source code classes (\ie, the classes that must be inspected in the QA process).


%An additional advantage of our approach is that we obtain precise and accurate structural information that has been fully processed by the compiler and embedded into the binaries.
%However, as shown in \fig{locfit}, this LOC estimator is usually inflated by a factor 3 with respect to the actual number of lines of source code, yet it correlates very well with it and we can, therefore, safely employ \LOC in our prediction model.  
%we obtain .  

%%Within the context of our approach, byte code and source code are proportionally equivalent (ref: figure with R2 near 1.0). That is, reading 10\% of the source code, is equivalent to processing 10\% of the byte code.

%to denote QA effort.

\begin{figure}[!h]
\begin{center}
%\includegraphics[width=0.72\columnwidth]{img/LinesOfCodeFit.pdf}
\includegraphics[width=0.78\columnwidth]{img/LinesOfCodeFit.pdf}
\end{center}
\caption{Lines of code (\LOC) extracted from byte code is a very strong approximation of the \LOC extracted directly from source code.}
\label{fig:locfit}
\end{figure}

%
%\begin{figure}[t]
%\begin{center}
%\includegraphics[width=0.9\columnwidth]{img/DataCollection.pdf}
%\end{center}
%\caption{We collect defect information from source control logs, while the metrics are collected from the compiled binaries.}
%\label{fig:dataset}
%\end{figure}
%
% ML:  We can keep this figure just as a visual illustration, no references to it needed.

The joined, activity-based, data sets are constructed as follows:

\be
\item From the PROMISE repository we fetch the bug information for release N per class.

\item We extract from the Helix repository the static code metrics, including \LOC, for release N per class.

\item Using the fully-qualified class name as key, both information is merged into the activity data set for release N per class.

\ee

%The data extraction approach that we take for our analysis is illustrated in Figure \ref{fig:dataset}. we need text showing how we\footnote{not enough detail. how to recognize setters? how to  recognize getters how to recognize that class A depends on class B (this is needed for "InDegrees") how is transitiveIndegree is computed please add those notes to Appendix.tex or to here}


\fig{defects} shows the distribution of defects seen in our classes. 
Usually, most classes have no defects, but in 10\% of cases, each class has more than 1 to 5 recorded defects.

\begin{figure}
\renewcommand{\baselinestretch}{0.7}
\footnotesize
\begin{center}
\begin{tabular}{l|c|ccccc}
    &              &\multicolumn{5}{c}{percentiles}\\%\cline{3-7}
	data&\# classes&10\%&30\%& 50\%& 70\% &90\%\\\hline
ant-1.3 & 125 &   0 &   0 &   0 &   0 &   1\\
ant-1.4 & 178&    0 &   0 &   0 &   0 &   1\\
ant-1.5 & 293&   0 &   0 &   0 &   0 &   1\\
ant-1.6 & 351&   0 &   0 &   0 &   0 &   2\\
ant-1.7 & 493&    0 &   0 &   0 &   0 &   2\\
ivy-1.4 & 241&   0 &   0 &   0 &   0 &   0\\
ivy-2 &352&    0 &   0 &   0 &   0 &   1\\
jedit-3.2 & 272&   0 &   0 &   0 &   1 &   4\\
jedit-4 & 306&   0 &   0 &   0 &   0 &   2\\
jedit-4.1 & 312&   0 &   0 &   0 &   0 &   2\\
jedit-4.2 & 367&   0 &   0 &   0 &   0 &   1\\
jedit-4.3 & 492&   0 &   0 &   0 &   0 &   0\\
lucene-2 & 195&   0 &   0 &   0 &   1 &   4\\
poi-2 & 314&   0 &   0 &   0 &   0 &   1\\
synapse-1 & 157&    0 &   0 &   0 &   0 &   0\\
synapse-1.1 & 222&   0 &   0 &   0 &   0 &   1\\
synapse-1.2 & 256&   0 &   0 &   0 &   1 &   2\\
velocity-1.6 & 229&   0 &   0 &   0 &   1 &   2\\
xalan-2.4 & 428&   0 &   0 &   0 &   0 &   1\\
xalan-2.5 &763&    0 &   0 &   0 &   1 &   2\\
xalan-2.6 &875&    0 &   0 &   0 &   1 &   2\\
xerces-1 & 162&   0 &   0 &   0 &   2 &   2\\
xerces-1.2 &438&    0 &   0 &   0 &   0 &   1\\
xerces-1.3 &452&    0 &   0 &   0 &   0 &   1\\
lucene-2.2 & 247&   0 &   0 &   1 &   2 &   4\\
lucene-2.4 & 428&   0 &   0 &   1 &   2 &   5\\
poi-1.5 &237&    0 &   0 &   1 &   1 &   4\\
poi-2.5 &384&    0 &   0 &   1 &   2 &   2\\
poi-3 & 442&   0 &   0 &   1 &   1 &   2\\
velocity-1.4 & 196&    0 &   1 &   1 &   1 &   2\\
velocity-1.5 & 214&   0 &   0 &   1 &   2 &   4\\
xalan-2.7 & 908&   1 &   1 &   1 &   1 &   2\\
xerces-1.4 & 329&   0 &   0 &   1 &   2 &   4\\
\end{tabular}
\end{center}
\caption{Percentile distributions, defects per class. For example, in xalan-2.7 the median
(50th percentile) defects per class is 1. In lucene-2.4, 10\% of classes have 5 defects
or more. In xalan-2.7, most classes have at least 1 defect.
The table is sorted by the median defects (see the 50\% percentile column).
}\label{fig:defects}
\end{figure}


%\subsection{Experimental Rig}
\subsection{Experimental Setup}

This study used a \emph{N-way cross-evaluation study}~\cite{witten05}. The data set is divided into $N=10$ 
buckets. For each bucket in the N-way, a predictor is learned on the nine of the buckets, then tested on the 
remaining bucket. These $N$ studies implement $N$ \emph{hold out} studies where a model is tested on data 
\emph{not} used in training.

To appreciate cross-validation, consider another approach called \emph{self-test} where the learned model
is assessed on the \emph{same} data that was used to create it. Self-tests are deprecated by the research 
community~\cite{witten05}. If the goal is to understand how well a defect predictor will work on future projects, 
it is best to assess the predictor via hold-out modules not used in the generation of that predictor. 

In the WEKA 3.7.3 implementation of the cross-val procedure used in this study, results are reported once for each 
test-instance as that instance appears in one of the $N$ hold-outs.\footnote{Note that prior to WEKA 3.7.2, the 
cross-val procedure\newline{\tt java -cp weka.jar \$learner -t file.arff}\\ incorrectly returns \emph{self-test} results.}
So a data set containing $C$ examples will generated $C$ predictions, regardless of the value of $N$ used for the 
number of hold-outs.

\subsection{Selection of Learners}\label{sec:learn}

As mentioned above, there are very many methods for converting static code measures into defect
predictors~\cite{mccabe76,lessmann09,taghi05a,taghi05,taghi00a,Harrison98a,
Briand99a,Briand00a,Nagappan06a,me07b,Turhan09a,me10d}. We adopted Holte's \emph{simplicity-first} 
heuristic~\cite{holte93} and applied a simple linear regression (LSR) algorithm available in WEKA~\cite{witten05}, 
with no pre-processing.

Note that WEKA's LSR tool uses a simple greedy \emph{back-select}, which is applied after the linear model has been 
generated. In that back-select, WEKA steps through all the attributes 
removing the one with the smallest standardized coefficient until no improvement is observed 
in the estimate of the model error given by the Akaike information 
criterion. As a consequence, some attributes may be absent from the final learned model. 

Initially, we planned to test various learners, feature extractors, instance selectors, and discretization methods 
(as we have done in the past~\cite{me07b,Turhan09a,me10d}). But our results were so
encouraging that there was little room for further improvement over simple LSR.

