% markus/InspectionOptimization.tex

\section{Inspection Optimization}
\label{inspect:sec}

%\markus{No evidence for claim: ``Larger classes tend to be more fault-prone than smaller classes. ~\cite{arisholm06}" page 6: 70\% code read 70\% defects located. With prediction we should be able to bring down the required effort (in terms of reading code), say, to 50\%. Structural properties of software are useful predictors of fault proneness ~\cite{arisholm06}. There is not much surprise in ~\cite{arisholm06}.}

% inspection success

How can our selected metrics help us in constructing an effective approach for fault localization in 
object-oriented software systems? To answer this question, we first need to develop a suitable 
\emph{success criterion} for defect prediction. For this paper, we adopt a proposal by Arisholm \& 
Briand~\cite{arisholm06} as follows:
\begin{quote}
\emph{If X\% of the classes are predicted to be defective, then the actual faults identified in those classes must 
account for more than X\% of all defects in the system being analyzed.
Otherwise, the costs of generating the defect predictor is not worth the effort.}
\end{quote}

In essence, this is \emph{inspection optimization} -- find some ordering to project artifacts such that humans have to read the \emph{least} code in order to discover the \emph{most} faults, which we model as follows:
\bi
\item
\emph{After} a data miner predicts a class is defective, \emph{then}  a secondary human team examines the code.
\item
This team correctly recognizes $\Delta$\% of the truly defective classes (and $\Delta=100\%$ 
means that the inspection team is perfect at its task and finds every defect present).
\item
A {\em good learner} is one that  finds the \emph{most} defective classes (measured in terms of probability of detection, \emph{pd}) in the \emph{smallest} classes (measured in terms of lines of code, \LOC).
\ei

\begin{figure}
\begin{center}
%\includegraphics[width=2.5in]{ant-14bif.pdf}
\includegraphics[width=0.96\columnwidth]{ant-14bif.pdf}
\end{center}
\caption{Percentage of defects found after sorting the code using different \emph{inspection ordering policies}.
Note that, in this case, developers were continually modifying a small number of very active classes handling 
complex interfacing tasks. Hence, for the blue curve, reading just this 1\% of the code found nearly a quarter
of the defects.}
\label{fig:auc}
\end{figure}

Inspection optimization can be visualized using \fig{auc} that illustrates three plausible
\emph{inspection ordering policies}:
\bi
\item
The blue $optimal$ policy combines knowledge of class size and the location of the actual defects. 
\item
The green $activity$ policy guesses defect locations using a defect predictor learned from the activity
measures.
\item
The red $baseline$ policy ignores defect counts and just sorts the classes in ascending order of size.
\ei

Each of these ordering policies  sorts the code base along the x-axis. The code is then inspected, left to right, across
that order, so that, by the end of the x-axis, we have read 100\% of the code. 
Along the way, we encounter classes containing y\% of defects (a.k.a. \emph{recall}).
A \emph{better} policy finds more defects sooner, that is, it yields a larger area under the curve of \%LOC-vs-recall.
In \fig{auc}, we note that 
that the green $activity$ policy does better than the red $baseline$ (and comes close, within 95\%, of the blue $optimal$).

%For Arisholm \& Briand to approve of a data miner for defect prediction, it must fall in the region \pd $>$ \loc of \fig{auc}. Here, we assume some \emph{ordering policy} that sorts the code base along the x-axis. The code is then inspected, left to right, across that order (so that, by the end of the x-axis, we have read 100\% of the code). Along the way, we encounter classes containing y\% of defects.

These three policies are defined by one equation modeling distance to some $utopia$ point of most defects
and smallest LOC:
{\small
\begin{equation}\label{eq:score}
0 \le \left( score(D_C,L_C,\alpha) = \frac{\sqrt{{\alpha}D_C^2 + (1-L_C)^2}}{\sqrt{\alpha + 1}}\right) \le 1
\end{equation}
}
Here, $D_C$ and $L_C$ are the number of defects and lines of code in class $C$ (normalized zero to one).
Also, $\alpha$ is a constant controlling the sorting.
At \mbox{$\alpha=0$}, we ignore defects and sort only on LOC. This implements the $baseline$ policy.
This baseline policy is the \emph{Koru ordering} advocated by researchers who argue that smaller
classes have a relatively higher density of errors~\cite{koru09,koru08,koru07}.
Note that if the $activity$ policy cannot out-perform $baseline$,
then our notion of activity is superfluous.
 
The other policies use $\alpha=1$. For the $activity$ policy, we:
\bi
\item Train a learner using the measures of \fig{metrics} without LOC,
\item Set $D_C$ via the learned model,
\item Sort using \eq{score}, $D_C$, LOC, and $\alpha=1$,
\item Calculate \fig{auc} and determine the area under the \%LOC-vs-recall curve.
\ei

%\noindent
The $optimal$ policy does the same, but sets $D_C$ using the historical defect logs.
Note that $optimal$ is different to $activity$ since the former knows exactly where the defects are, whereas
the latter must guess the defect locations using the learned model.

In practice, the $optimal$ policy is impossible to apply since it implies that we were to know the number of defects 
\emph{before} the classes would be inspected. However, it is the theoretical upper-bound on performance
for inspection optimization. Hence, we report $activity$ and $baseline$ performances
as a ratio of the area under the curve of $optimal$.

This ratio calculation has another advantage. Note that the $\Delta$ effectiveness of the secondary human
inspection team is same, regardless of the oracle that sorts the code. 
Hence, in the ratio calculation,  $\Delta$ cancels out and we can ignore it from our analysis.
