% This is "sig-alternate.tex" V1.9 April 2009
% This file should be compiled with V2.4 of "sig-alternate.cls" April 2009
%
% This example file demonstrates the use of the 'sig-alternate.cls'
% V2.4 LaTeX2e document class file. It is for those submitting
% articles to ACM Conference Proceedings WHO DO NOT WISH TO
% STRICTLY ADHERE TO THE SIGS (PUBS-BOARD-ENDORSED) STYLE.
% The 'sig-alternate.cls' file will produce a similar-looking,
% albeit, 'tighter' paper resulting in, invariably, fewer pages.
%
% ----------------------------------------------------------------------------------------------------------------
% This .tex file (and associated .cls V2.4) produces:
%       1) The Permission Statement
%       2) The Conference (location) Info information
%       3) The Copyright Line with ACM data
%       4) NO page numbers
%
% as against the acm_proc_article-sp.cls file which
% DOES NOT produce 1) thru' 3) above.
%
% Using 'sig-alternate.cls' you have control, however, from within
% the source .tex file, over both the CopyrightYear
% (defaulted to 200X) and the ACM Copyright Data
% (defaulted to X-XXXXX-XX-X/XX/XX).
% e.g.
% \CopyrightYear{2007} will cause 2007 to appear in the copyright line.
% \crdata{0-12345-67-8/90/12} will cause 0-12345-67-8/90/12 to appear in the copyright line.
%
% ---------------------------------------------------------------------------------------------------------------
% This .tex source is an example which *does* use
% the .bib file (from which the .bbl file % is produced).
% REMEMBER HOWEVER: After having produced the .bbl file,
% and prior to final submission, you *NEED* to 'insert'
% your .bbl file into your source .tex file so as to provide
% ONE 'self-contained' source file.
%
% ================= IF YOU HAVE QUESTIONS =======================
% Questions regarding the SIGS styles, SIGS policies and
% procedures, Conferences etc. should be sent to
% Adrienne Griscti (griscti@acm.org)
%
% Technical questions _only_ to
% Gerald Murray (murray@hq.acm.org)
% ===============================================================
%
% For tracking purposes - this is V1.9 - April 2009

\documentclass{sig-alternate}
\usepackage{cite,times}
\usepackage[usenames,dvipsnames]{color}
\usepackage{url}
\usepackage{tim/tricks}
\usepackage{raj/tricks}
\usepackage{markus/tricks}

\begin{document}
%
% --- Author Metadata here ---
\conferenceinfo{MSR}{'11 Honolulu,  Hawaii USA}
%\CopyrightYear{2007} % Allows default copyright year (20XX) to be over-ridden - IF NEED BE.
%\crdata{0-12345-67-8/90/01}  % Allows default copyright data (0-89791-88-6/97/05) to be over-ridden - IF NEED BE.
% --- End of Author Metadata ---

\title{'Popularity' is (almost) a Perfect Predictor for Defects}
%
% You need the command \numberofauthors to handle the 'placement
% and alignment' of the authors beneath the title.
%
% For aesthetic reasons, we recommend 'three authors at a time'
% i.e. three 'name/affiliation blocks' be placed beneath the title.
%
% NOTE: You are NOT restricted in how many 'rows' of
% "name/affiliations" may appear. We just ask that you restrict
% the number of 'columns' to three.
%
% Because of the available 'opening page real-estate'
% we ask you to refrain from putting more than six authors
% (two rows with three columns) beneath the article title.
% More than six makes the first-page appear very cluttered indeed.
%
% Use the \alignauthor commands to handle the names
% and affiliations for an 'aesthetic maximum' of six authors.
% Add names, affiliations, addresses for
% the seventh etc. author(s) as the argument for the
% \additionalauthors command.
% These 'additional authors' will be output/set for you
% without further effort on your part as the last section in
% the body of your article BEFORE References or any Appendices.

\numberofauthors{2} %  in this sample file, there are a *total*
% of EIGHT authors. SIX appear on the 'first-page' (for formatting
% reasons) and the remaining two appear in the \additionalauthors section.
%
\author{
% You can go ahead and credit any number of authors here,
% e.g. one 'row of three' or two rows (consisting of one row of three
% and a second row of one, two or three).
%
% The command \alignauthor (no curly braces needed) should
% precede each author name, affiliation/snail-mail address and
% e-mail address. Additionally, tag each line of
% affiliation/address with \affaddr, and tag the
% e-mail address with \email.
%
% 1st. author
\alignauthor
Markus Lumpe, Rajesh Vasa\\
       \affaddr{Faculty of Information \& Communication}
       \affaddr{Swinburn University of Technology}\\
       \affaddr{Melbourne, Australia}\\
       \email{mlumpe@swin.edu.au, rajesh.vasa@gmail.com}
\alignauthor
Tim Menzies\\
       \affaddr{Lane Department of Com. Sci. \& Elec. Eng.}\\
       \affaddr{West Virginia University}\\
       \affaddr{Morgantown, USA}\\
       \email{tim@menzies.us}
}
% There's nothing stopping you putting the seventh, eighth, etc.
% author on the opening page (as the 'third row') but we ask,
% for aesthetic reasons that you place these 'additional authors'
% in the \additional authors block, viz.
\date{24 Jan 2011}
% Just remember to make sure that the TOTAL number of authors
% is the number that will appear on the first page PLUS the
% number that will appear in the \additionalauthors section.

\maketitle
\begin{abstract}
BACKGROUND:
There is much new and successful
research on social metrics and defect prediction. 
Yet our open source repositories often lack the
detailed developer information required from a social
metrics analysis.
\\
AIM:
Have social metrics
work superseded prior work on defect prediction from
static code measures? 
\\
METHOD:
Quasi social metrics (about the activities of individuals using a repository)
are extracted
open source repositories (which lack details about individual developers).
The value of these  metrics are assessed in context of optimizing
software inspections (reading the fewest lines of code to find the most defects).
\\
RESULTS:
For the tasking of inspection
optimization,
the quasi social metrics perform nearly
as well
(usually, within 3\%) 
of the theoretical upper bound on the performance  
of any metric set.
\\
CONCLUSION:
Research of defect prediction from
static code measures have not been superseded by
social metrics.
There is still much to be usefully discovered in software repositories,
even if those repositories lack detailed developer information.
\end{abstract}

% A category with the (minimum) three required fields
%A category including the fourth, optional field follows...
\category{D.2.8}{Software Engineering}{Metrics}[complexity measures, performance measures]

\section{Introduction}
One of the remarkable recent discoveries in mining software repositories
is that {\em social metrics} that model the sociology of the programmers working on the code
can be an effective predictor for defect injection and removal~\cite{bird09,bird09a,nach08}.
For example, Guo et al.~\cite{Guo2010}
report that for the purposes of predicting if a bug will
get fixed, a vital feature is the reputation of
the developer who reported the bug.

In the standard view, defect prediction via social metrics requires detailed knowledge of developers.
For example, in the data used for mining, there appears a name (or an anonymous id) of a developer
plus some meta-knowledge about their social context
(e.g. their reputation, who they email, who they are
co-located with, etc). 

There exist many datasets without such detailed developer knowledge. For example,
in our work on the Helix project, we have studied 50 multi-year large open source projects under active development.
While many developers work on these projects, the repositories storing the code are very weak
sources of social metrocs.

This paper argues that if a repository lacks detailed developer knowledge, then some surrogate measures
of social activity can still be accessed. In particular, it is possible to infer which parts of the code
are most {\em popular}; i.e. are the focus of much of the attention of the developers\footnote{rajesh: is this
the definition? and for oo? need some metrics on the ratio of popular classes}. Further, and this
is the core message of this paper:
\begin{quote}
Metrics that focus on code popularity predicts  are good predictors for defects.
\end{quote}
Popularity can lead to defect discovery in two ways:
\bi
\item {\em Defect injection:} As developers work with the popular classes, they make the occasional mistake.
Some of these mistakes result in code defects. Since developers work on popular classes more than other classes,
then most developer defects accumulate in the popular classes.
\item {\em Defect discovery:} Since developers work mostly on popular classes, they are most likely to uncover
defects in those  classes.
\ei
As shown below, these popularity metrics are very simple to collect. For example, one popularity metrics
we rely on for defect prediction is {\em number of getter methods} that developers have added to a class. One advantage
of using such simple metrics is that it is not difficult to build parsers that can automatically extract information
on popularity.

A disadvantage with using such simple metrics is that they may be {\em too} simple. As we define them below,
the popularity metrics
lack much of the information
seen in the standard social metrics. For example, while our popularity metrics let us defect {\em what} classes
have been changed, they are uninformative about {\em who} actually editted any particular class.

Therefore, the task of this paper is to assess the value of the popularity  metrics for defect prediction.
Our pre-experimental expectation was that popularity would be a {\em lightweight} metric; i.e. quick to collect,
simple to apply, moderately useful, but out-performed by more heavyweight defect prediction methods.
This turned out not to be the case.  In 31 experiments with open source data, we compared the performance of the popularity metrics
against a theoretical upper bound on defect performance. To compute that upper bound, we adopt the defect inspection bias
proposed by 
 indicator for defects, 

lets us see which classes were most
editted, 
Two recent, and separate, approaches to
mining software repositories
focus either or (a)~static code measures for defect prediction
or (b)~``social metrics'' for defect analysis:
\bi
\item
Static code defect predictors can automatically process thousands of modules in a matter of seconds. 
Alternative methods such as
manual inspections are much slower (8 to 20 LOC per minute)~\cite{me07b}.
They have been successfully applied in 
software companies in many countries including
the United States~\cite{ism06}  and Turkey~\cite{tosun09,tosun10}.
Hence, they are widely-used: 
elsewhere~\cite{me10a}, we have reported numerous papers that explore their use,
dating back as far back as 1990~\cite{porter90}. 
\item
Numerous  recent results (e.g.~\cite{bird09,bird09a,nach08})
now show that understanding
the patterns of social interaction between developers can be insightful 
for defect prediction. \ei
Social metrics are currently very prominent in the literature.
Their popularity is such that 
researchers 
need now
justify why they continue to work on something as seemingly
obsolete 
as static code measures (as witnessed by one of us in
a recent review of their rejected
paper from a senior research journal).

This article argues that continued research into static code
features is useful and insightful. Using static code features,
it is possible to make novel 
and exciting conclusions about defects,  without detailed
developer information. In fact, it will be shown that some
developer information can be {\em inferred} from standard
repositories.

More specifically,
we distinguish two
classes of social metrics:
\bi
\item 
The {\em standard social} metrics
as used by, say, Guo, Nagappan and 
Bird et al.~\cite{Guo2010,bird09,bird09a,nach08};
\item
The {\em quasi social} metrics explored by this paper.
\ei
The difference between them is as follows. In the {\em standard}
sOn the other hand, in the {\em quasi} set,
there is no information about individuals and their social
context.  Rather, we infer social patterns of
developers via
static code measures.

Further, and more importantly,
we show that for the purposes of optimizing software inspections,
our quasi metrics {\em are usually perfect}: in the majority case,
they
perform
very close (within 3\%) of a theoretical upper bound.
That is, our quasi metrics are 
{\em good} and no other set of metrics {\em may ever be better}.
The two caveats on this claim are:
\bi
\item
``Good'' and ``better'' are defined
in terms of one particular task (optimizing software inspections).
For other tasks, it is possible other metrics could
out-perform our quasi set.
\item We have only assessed this result on 
twelve data sets.  
\ei
(These caveats
are hardly restrictive.
A dozen data sets is larger than
the evaluation corpus used in many other data sets. Also,
the task discussed here- inspection optimization- 
is one advocated widely in the literature;
e.g. by Arisholm and Briand~\cite{arisholm06}.)

From this study, we can conclude
there is still much to be usefully discovered in software repositories,
{\em even if those repositories lack detailed developer information}.
Static code measures have not been superseded by
social metrics.
Rather than take a position that static code measure are ``bad''
and social metrics are ``better'', this paper suggests that:
\bi
\item
Static code measures can take on multiple roles;
\item
In some of those roles, they are act as quasi-social metrics; 
\item
In that quasi-role, they can be remarkably successful.
\ei
Hence, in future, we would propose continuing extensive research
into {\em both} static code features {\em and} social metrics, without
any expectation that one will somehow subsume and retire the other.

The rest of this paper is structured as follows.

\section{Defect Prediction}
When building
a software system, engineers build and combine
software ``modules'';
i.e.
a small unit of compilation which might be
a
function in ``C'' or a class in ``JAVA''.
Typically, one {\em project} contains hundreds to tens of thousands
of {\em modules}.

We assume that some quality assurance (QA) technique is applied
to assess module quality.
Those techniques include inspections, unit tests, static source code analyzers, etc.
A record of the results of this QA is a {\em defect log}. 
{\em Defect predictors} can be learned from these historical logs,
if the logs contain tables of data. In those tables,
rows describe data from one module.
Table
columns describe static code features such as lines of code
measures, the McCabe measures, the Halstead measures and (for OO systems), the number of methods per class,
the number of children per class, etc. One column is added for 
``{\em bugs}''; i.e.  a count of how many defects where found
during the inspection process.

For these data sets, the data mining goal is to learn a 
predictor for the number of
$defective$ modules from past projects that can be
applied to future projects.
Such defect predictors can be used to focus the
QA budgets.
This is an important task since,
during development, developers {\em skew} their limited quality assurance (QA)
budgets
towards artifacts they believe most require extra QA.
For example, it is common at NASA to focus QA more on the on-board guidance system than
the ground-based database storing scientific data collected from a satellite.

This skewing process can introduce an inappropriate bias to QA.
If  the QA activities concentrate on  project
artifacts, say {\em A,B,C,D}, then that leaves {\em blind spots} in
{\em E,F,G,H,I,...}.
Blind spots can compromise high assurance
software. Leveson remarks that in modern complex systems, unsafe
operations often result from an unstudied interaction between
components~\cite{leven95}. For example, Lutz and
Mikulski~\cite{lutz03} found a blind spot in NASA deep-space
missions: most of the mission critical {\em in-flight} anomalies
resulted from errors in {\em ground software} that fails to
correctly collect in-flight data.

To avoid blind spots, one option is to rigorously assess all aspects
of all  software modules. But this is
impractical. Software project budgets are finite and QA
effectiveness increases with QA effort:
\bi
\item
 A {\em linear} increase in
the confidence $C$ of finding faults takes {\em
exponentially} more effort. For example, to detect one-in-a-thousand
module faults, moving $C$ from 90\% to 94\% to 98\% takes 2301,
2812, and 3910 black box tests (respectively)\footnote{A randomly
selected input will find a fault with probability $x$.
Voas observes~\cite{voas95} that after $N$ random black-box tests,
the chance of the inputs not revealing any fault is $(1-x)^N$.
Hence, the chance $C$ of seeing the fault is $1-(1-x)^N$ which can
be rearranged to
 $N(C,x)=\frac{log(1 -
C)}{log(1-x)}$. For example, $N(0.90,10^{-3})=2301$.}.
\item
The
infamous state space explosion problem imposes
strict limits on how much a system can be explored
via automatic formal methods~\cite{me00q}.
Lowry et.al.~\cite{lowrey98} and Menzies and Cukic~\cite{me99q}
offer numerous other examples
where assessment effectiveness is exponential on effort.
\ei
Exponential costs quickly exhausts finite QA resources.
Hence, blind spots can't be avoided and must be managed. Standard
practice is to apply the best available assessment methods on the
sections of the program that the best available domain knowledge
declares is the most critical.  We endorse this approach.  Clearly,
critical sections require the best known assessment
methods, in hope of minimizing the risk of safety or mission
critical failure occurring post deployment. However, this focus on
certain sections can blind us to defects in other areas which,
through interactions, may cause similarly critical failures.
Therefore, the standard practice should be augmented with a {\em
lightweight sampling policy} like defect predictors
that (a)~explores the rest of the
software and (b)~raises an alert on parts of the software that
appear problematic.  This sampling approach is incomplete by
definition. Nevertheless, it is the only option when
resource limits block
complete assessment.

Compared to certain reports of standard industrial practice, defect
predictors are better than humans are identifying which modules will
require further QA. Previously we have reported studies where defect
predictors had probabilities of detection (recall) and probabilities
of false alarm (pf) of (recall,pf)=(71\%,25\%), on
average~\cite{me07b}. These values are higher than known results
from manual inspection methods. For example, a panel at {\em IEEE
Metrics 2002}~\cite{shu02} concluded that manual software reviews
can find ${\approx}60\%$ of defects\footnote{That panel supported
neither Fagan's claim~\cite{fagan86} that inspections can find 95\%
of defects before testing or Shull's claim that specialized directed
inspection methods can catch 35\% more defects that other
methods~\cite{shull00a}.}. In other work, Raffo found that the
defect detection capability of industrial review methods can vary
from
\[
Recall=TR(35, 50, 65)\%\]
for full Fagan inspections~\cite{fagan76} to
\[
Recall=TR(13, 21, 30)\%\]
for less-structured inspections~\cite{raffo05}
(note: ``$TR(a,b,c)$'' is a triangular distribution with
min/mode/max of $a,b,c$).

Defect prediction can also be faster than human-intensive methods.
Static code features can be automatically and cheaply extracted from
source code, even for very large systems~\cite{nagappan05}. By
contrast, other methods such as manual code reviews are
labor-intensive.  Depending on the review methods 8 to 20 LOC/minute
can be inspected and this effort repeats for all members of the
review team, which can be as large as four or six~\cite{me02f}.

Also, there are industrial reports where this style of defect prediction
was found pragmatically useful by industry:
\bi
\item Defect predicting technology has been commercialized in commercial
tools like
  {\em Predictive}~\cite{ism06}.
One
company used these tools to manage  safety critical
software for  fighter aircraft
  (the software controlled a
 lithium ion battery, which can overcharge and possibly
  explode).  After applying a more expensive tool for structural
  code coverage was applied,  the company ran Predictive on the same code base.
  Predictive produced consistent results with the more expensive tools
 while
being able to faster process a larger code base than the
more expensive tool~\cite{ism06}.
\item
 Defect predictors
developed at NASA ~\cite{me07b}
have been applied to software development companies in
another country (in Turkey).
When inspection teams focused on the modules that trigger the defect predictors,
they found up to 70\% of the defects using
40\% of the effort (measured in staff hours)~\cite{tosun09}.
\item
A  subsequent study on the Turkish software
compared how much code needs to be
inspected using random selection vs selection via our
defect predictors. Using random testing,
87\% of the files
would have to be inspected in order to detect 87\% of the
defects.
However, if the inspection process was restricted to the
25\% of the files that trigger the defect
predictors, then 88\% of the defects could be found.
That is, the same level of defect detection (after inspection)
can be achieved using $\frac{87-25}{87}=71\%$ less effort~\cite{tosun10}.
\ei
\section{Inspection Optimization}
In order to assess the quasi-social metrics proposed in this paper, we
adopt the
Arisholm \& Briand~\cite{arisholm06} 
success criteria for  defect
prediction:
\bi
\item
A budget-conscious team wants to know that if X\% of the classes
are predicted to be defective, then those classes contain more than X\%
of the defects.  
\item
Otherwise, they might argue, the cost of generating the
defect predictor is not worth the effort.
\ei
In essence, their concept is {\em inspection optimization}; i.e. find some
ordering to project artifacts such that humans have to read the {\em least} to
discover the {\em most}.

To operationalize this concept,  we assume that:
\bi
\item
{\em After} a data miner predicts a class is defective, {\em then} 
a secondary human team examines the code.
\item
This team correctly recognize some subset $\Delta$  of the truly defective classes,
(and
$\Delta=1$ means that the inspection teams are perfect at their task).
\item
Our goal is to 
 find   learners that find the $most$ number of defective modules (measured in terms of probability of detection, $pd$)
in the $smallest$ number of modules (measured in terms of loc).
\ei
For Arisholm \& Briand to approve of a data miner, it must fall in the region
$pd > loc$ of
\fig{auc}. To generate this figure, we assume some {\em ordering policy} that sorts
the code base along the x-axis. The code is then inspected, left to right, across that order
(so that, by the end of the x-axis, we have read 100\% of the code).
Along the way, we encounter classes containing y\% of defects.

\begin{figure}
\begin{center}
\includegraphics[width=3in]{ant-14bif.pdf}
\end{center}
\caption{Visualizing inspection optimization. Better inspection policies
are those that rise higher above the diagonal of \mbox{$defects\; found =loc\; read$}.
The ratio of the area under the two curves is $\frac{learning}{optimal}=95\%$. }\label{fig:auc}
\end{figure}


What ordering policy should be applied? To satisfy 
Arisholm \& Briand~\cite{arisholm06}, we sort we normalize the each class's defects $d$
and lines of code $l$ to the range 0..1 as follows:
\[
D=\frac{d - min(d)}{max(d) - min(d)} \;\;\;
L=\frac{l - min(l)}{max(l) - min(l)}
\]
We then sort each class favoring those with most defects and smallest lines of code.
Formally, this can be modeled as the Euclidean distance to max defects and min lines of code:
\begin{equation}\label{eq:score}
0 \le \left( score(D,L) = \frac{\sqrt{aD^2 + b(1-L)^2}}{\sqrt{a+b}}\right) \le 1
\end{equation}
Here, $(a,b)$ are tuning constants that can control how much we emphasize defects or lines of code.
Using \eq{score}, if we inspect the higher scoring classes first, then we
are reading the smaller, most defective classes {\em before} reading the larger, less defective classes.

It is good practice to
compare the performance of some technique against 
theoretical {\em upper} bound~\cite{cohen95}.
Using \eq{score},
we define 
such a maximum upper bound on our performance as follows.
Let $D_a$ be the actual number of defects seen in the historical log.
An  {\em optimal} oracle 
inspects classes according to the sort order defined by $score(D_a,L)$.

In practice, this optimal oracle is impossible to apply 
since this would imply known the number of defects $before$
the classes were inspected. What is practical, however, is to
apply some data mining technique 
to learn $D_p$, a prediction generated by (say) linear regression from
the historical data (taking care to ensure that the class to be predicted {\em does not} appear in
the historical data set). 
This  {\em learning} oracle 
inspects classes according to the sort order defined by $score(D_p,L)$.

The results of applying the optimal and learning oracle are shown as the red and green lines (respectively) in
\fig{auc}. 
Note that:
\bi
\item
The y-axis of this figure assumes $\Delta=1$;
i.e. inspection teams  correctly recognizes all defective modules.
\item
The $\Delta$ effectiveness of the secondary human inspection team  is same, 
regardless of the oracle that sorts the code. Hence, it we divide
the two areas under the two curves of \fig{auc}, then $\Delta$
cancels out and we can ignore it from our analysis.
\ei
Another advantage of dividing the areas under the curves of \fig{auc}
is that we an express the effectiveness of the learning oracle
as a ratio of the optimal oracle. For example, in \fig{auc},
the areas under the curves $\frac{learning}{oracle}=95\%$; i.e.
in this case, the predictions of the learning oracle as nearly
as effective (within 5\%) of the optimal.

\section{Methods}
The rest of this paper  finds the $\frac{learning}{oracle}$ ratio of
our quasi social metrics for 12 data sets.

These data sets were selected as follows.

XXX

\fig{defects} shows the distribution of defects seen in our classes.
Usually, most classes have no defects;
but in 10\% of cases,
each class has more than 1 to 5 recorded defects.

\begin{figure}
\begin{tabular}{l|r|rrrrr}
    &              &\multicolumn{5}{c}{percentiles}\\\cline{3-7}
	data&\# classes&10\%&30\%& 50\%& 70\% &90\%\\\hline
ant-1.3 & 125 &   0.0 &   0.0 &   0.0 &   0.0 &   1.0\\
ant-1.4 & 178&    0.0 &   0.0 &   0.0 &   0.0 &   1.0\\
ant-1.5 & 293&   0.0 &   0.0 &   0.0 &   0.0 &   1.0\\
ant-1.6 & 351&   0.0 &   0.0 &   0.0 &   0.0 &   2.0\\
ant-1.7 & 493&    0.0 &   0.0 &   0.0 &   0.0 &   2.0\\
ivy-1.4 & 241&   0.0 &   0.0 &   0.0 &   0.0 &   0.0\\
ivy-2.0 &352&    0.0 &   0.0 &   0.0 &   0.0 &   1.0\\
jedit-3.2 & 272&   0.0 &   0.0 &   0.0 &   1.0 &   4.0\\
jedit-4.0 & 306&   0.0 &   0.0 &   0.0 &   0.0 &   2.0\\
jedit-4.1 & 312&   0.0 &   0.0 &   0.0 &   0.0 &   2.0\\
jedit-4.2 & 367&   0.0 &   0.0 &   0.0 &   0.0 &   1.0\\
jedit-4.3 & 492&   0.0 &   0.0 &   0.0 &   0.0 &   0.0\\
lucene-2.0 & 195&   0.0 &   0.0 &   0.0 &   1.0 &   4.0\\
poi-2.0 & 314&   0.0 &   0.0 &   0.0 &   0.0 &   1.0\\
synapse-1.0 & 157&    0.0 &   0.0 &   0.0 &   0.0 &   0.0\\
synapse-1.1 & 222&   0.0 &   0.0 &   0.0 &   0.0 &   1.0\\
synapse-1.2 & 256&   0.0 &   0.0 &   0.0 &   1.0 &   2.0\\
velocity-1.6 & 229&   0.0 &   0.0 &   0.0 &   1.0 &   2.0\\
xalan-2.4 & 428&   0.0 &   0.0 &   0.0 &   0.0 &   1.0\\
xalan-2.5 &763&    0.0 &   0.0 &   0.0 &   1.0 &   2.0\\
xalan-2.6 &875&    0.0 &   0.0 &   0.0 &   1.0 &   2.0\\
xerces-1.0 & 162&   0.0 &   0.0 &   0.0 &   2.0 &   2.0\\
xerces-1.2 &438&    0.0 &   0.0 &   0.0 &   0.0 &   1.0\\
xerces-1.3 &452&    0.0 &   0.0 &   0.0 &   0.0 &   1.0\\
lucene-2.2 & 247&   0.0 &   0.0 &   1.0 &   2.0 &   4.0\\
lucene-2.4 & 428&   0.0 &   0.0 &   1.0 &   2.0 &   5.0\\
poi-1.5 &237&    0.0 &   0.0 &   1.0 &   1.0 &   4.0\\
poi-2.5 &384&    0.0 &   0.0 &   1.0 &   2.0 &   2.0\\
poi-3.0 & 442&   0.0 &   0.0 &   1.0 &   1.0 &   2.0\\
velocity-1.4 & 196&    0.0 &   1.0 &   1.0 &   1.0 &   2.0\\
velocity-1.5 & 214&   0.0 &   0.0 &   1.0 &   2.0 &   4.0\\
xalan-2.7 & 908&   1.0 &   1.0 &   1.0 &   1.0 &   2.0\\
xerces-1.4 & 329&   0.0 &   0.0 &   1.0 &   2.0 &   4.0\\
\end{tabular}
\caption{Percentile distributions of number of defects per class. For example: in xalan-2.7, the median
number (50th percentile) of defects per class is 1; in lucene-2.4, 10\% of classes have 5 defects
or more; in xalan-2.7, most classes have at least 1 defect.
}\label{fig:defects}
\end{figure}

\input{raj/data}
\input{markus/metrics}


%
%\begin{figure*}
%\noindent
%\includegraphics[width=2in]{img/ant-13.pdf}        
%\includegraphics[width=2in]{img/ant-14.pdf}     
%\includegraphics[width=2in]{img/ant-15.pdf}     
%
%\noindent
%\includegraphics[width=2in]{img/ant-16.pdf}       
%\includegraphics[width=2in]{img/ant-17.pdf} 
%\includegraphics[width=2in]{img/lucene-20.pdf} 
%
%\noindent
%\includegraphics[width=2in]{img/lucene-22.pdf}
%\includegraphics[width=2in]{img/lucene-24.pdf}
%\includegraphics[width=2in]{img/xalan-24.pdf}
%
%\noindent
%\includegraphics[width=2in]{img/xalan-25.pdf}
%\includegraphics[width=2in]{img/xalan-26.pdf}
%\includegraphics[width=2in]{img/xalan-27.pdf}
%\caption{Results}
%\end{figure*}
%
\section{Results}
\subsection{Does ``Popularity'' Predict for Defects?}

\begin{figure}
\begin{center}
\begin{tabular}{l|rrrrr}
    &              \multicolumn{5}{c}{percentiles}\\\cline{2-6}
    data         &        10\%&  30\% &   50\% & 70\%  &  90\%\\\hline
lucene-2.4 &  -2.0 &  -0.9 &  -0.3 &   0.4 &   2.6 \\ 
velocity-1.6 &  -1.4 &  -0.6 &  -0.3 &   0.0 &   1.4 \\ 
xerces-1.0 &  -1.1 &  -0.6 &  -0.3 &   0.7 &   1.0 \\ 
ant-1.4 &  -0.4 &  -0.3 &  -0.2 &  -0.1 &   0.8 \\ 
lucene-2.0 &  -2.0 &  -0.8 &  -0.2 &   0.1 &   1.7 \\ 
poi-1.5 &  -1.6 &  -0.9 &  -0.2 &   0.0 &   2.3 \\ 
synapse-1.2 &  -0.8 &  -0.4 &  -0.2 &   0.0 &   1.0 \\ 
xalan-2.5 &  -0.9 &  -0.5 &  -0.2 &   0.5 &   0.8 \\ 
xalan-2.6 &  -0.9 &  -0.5 &  -0.2 &   0.4 &   1.2 \\ 
xalan-2.7 &  -0.5 &  -0.3 &  -0.2 &   0.1 &   0.7 \\ 
xerces-1.2 &  -0.5 &  -0.2 &  -0.2 &  -0.0 &   0.9 \\ 
xerces-1.4 &  -2.2 &  -0.4 &  -0.2 &   0.4 &   0.8 \\ 
ant-1.3 &  -0.5 &  -0.3 &  -0.1 &   0.0 &   0.6 \\ 
ant-1.7 &  -0.8 &  -0.3 &  -0.1 &   0.0 &   0.8 \\ 
ivy-2.0 &  -0.3 &  -0.1 &  -0.1 &   0.0 &   0.3 \\ 
lucene-2.2 &  -2.5 &  -0.8 &  -0.1 &   0.4 &   2.0 \\ 
poi-2.0 &  -0.2 &  -0.1 &  -0.1 &  -0.1 &   0.7 \\ 
poi-3.0 &  -1.1 &  -0.4 &  -0.1 &   0.0 &   1.0 \\ 
synapse-1.0 &  -0.3 &  -0.2 &  -0.1 &   0.0 &   0.0 \\ 
synapse-1.1 &  -0.7 &  -0.4 &  -0.1 &   0.0 &   0.9 \\ 
velocity-1.5 &  -1.5 &  -0.7 &  -0.1 &   0.3 &   1.5 \\ 
xalan-2.4 &  -0.5 &  -0.2 &  -0.1 &   0.0 &   0.7 \\ 
ant-1.5 &  -0.3 &  -0.1 &  -0.0 &   0.0 &   0.3 \\ 
ant-1.6 &  -0.8 &  -0.3 &  -0.0 &   0.0 &   0.9 \\ 
ivy-1.4 &  -0.2 &  -0.1 &  -0.0 &   0.0 &   0.0 \\ 
jedit-3.2 &  -2.3 &  -0.8 &   0.0 &   0.0 &   1.8 \\ 
jedit-4.0 &  -1.3 &  -0.5 &   0.0 &   0.0 &   1.0 \\ 
jedit-4.1 &  -1.1 &  -0.4 &  -0.0 &   0.0 &   1.0 \\ 
jedit-4.2 &  -0.6 &  -0.2 &  -0.0 &   0.0 &   0.3 \\ 
jedit-4.3 &  -0.1 &  -0.0 &  -0.0 &  -0.0 &   0.0 \\ 
poi-2.5 &  -1.3 &  -0.6 &  -0.0 &   0.7 &   0.9 \\ 
velocity-1.4 &  -1.2 &  -0.2 &  -0.0 &   0.1 &   1.0 \\ 
xerces-1.3 &  -1.4 &  -0.2 &  -0.0 &   0.0 &   0.7  
\end{tabular}
\end{center}
\caption{Percentile distributions of $actual - predicted$ number of defects per class. For example,
the median (50th percentile) value of $actual - predicted$ is -0.3 to 0.}\label{fig:ar}
\end{figure}

\fig{ar} shows the distribution of $actual - predicted$ defects
for our classes where $actual$ comes from historical logs and $predicted$ comes from
linear regression over some training set).
Note that, in the median case, the predictions are very close to actuals (-0 to -0.4).
Hence:
\bi
\item
 The quasi-social metrics that predict for
class popularity are adequate defect predictors.
\item
Our thesis that {\em usage predicts for defects} is supported.
\ei
\subsection{Effectiveness}
\fig{pm} shows the effectiveness of the popularity metrics,
expressed as the ratio $\frac{learned}{optimal}$.  Note
the median value of that figure: our popularity-based metrics perform
within 3\% of the optimal. Hence, we say that not only are quasi-social popularity
metrics
 good predictors, but also that  the addition of the standard social metrics
(containing detailed developer knowledge) while add very little extra value.
\newcommand{\wwidth}{2.5mm}
\begin{figure}
\begin{center}
\begin{tabular}{lrlr}
synapse-1.0 & 74 & \textcolor[named]{Gray}{\rule{24mm}{\wwidth}}\\
ant-1.3 & 77 & \textcolor[named]{Gray}{\rule{25mm}{\wwidth}}\\
poi-2.0 & 78 & \textcolor[named]{Gray}{\rule{25mm}{\wwidth}}\\
jedit-4.3 & 81 & \textcolor[named]{Gray}{\rule{26mm}{\wwidth}}\\
ivy-2.0 & 85 & \textcolor[named]{Gray}{\rule{28mm}{\wwidth}}\\
ivy-1.4 & 89 & \textcolor[named]{Gray}{\rule{29mm}{\wwidth}}\\
ant-1.5 & 90 & \textcolor[named]{Gray}{\rule{29mm}{\wwidth}}\\
velocity-1.6 & 91 & \textcolor[named]{Gray}{\rule{30mm}{\wwidth}}\\
synapse-1.1 & 92 & \textcolor[named]{Gray}{\rule{30mm}{\wwidth}}\\
poi-3.0 & 93 & \textcolor[named]{Gray}{\rule{30mm}{\wwidth}}\\
ant-1.7 & 94 & \textcolor[named]{Gray}{\rule{31mm}{\wwidth}}\\
lucene-2.0 & 94 & \textcolor[named]{Gray}{\rule{31mm}{\wwidth}}\\
velocity-1.5 & 94 & \textcolor[named]{Gray}{\rule{31mm}{\wwidth}}\\
xerces-1.3 & 94 & \textcolor[named]{Gray}{\rule{31mm}{\wwidth}}\\
ant-1.4 & 95 & \textcolor[named]{Gray}{\rule{31mm}{\wwidth}}\\
poi-1.5 & 96 & \textcolor[named]{Gray}{\rule{31mm}{\wwidth}}\\
poi-2.5 & 96 & \textcolor[named]{Gray}{\rule{31mm}{\wwidth}}\\
xerces-1.4 & 96 & \textcolor[named]{Gray}{\rule{31mm}{\wwidth}}\\
synapse-1.2 & 97 & \textcolor[named]{Gray}{\rule{32mm}{\wwidth}}\\
xalan-2.4 & 97 & \textcolor[named]{Gray}{\rule{32mm}{\wwidth}}\\
xerces-1.0 & 97 & \textcolor[named]{Gray}{\rule{32mm}{\wwidth}}\\
xerces-1.2 & 97 & \textcolor[named]{Gray}{\rule{32mm}{\wwidth}}\\
xalan-2.5 & 98 & \textcolor[named]{Gray}{\rule{32mm}{\wwidth}}\\
xalan-2.6 & 98 & \textcolor[named]{Gray}{\rule{32mm}{\wwidth}}\\
lucene-2.4 & 99 & \textcolor[named]{Gray}{\rule{32mm}{\wwidth}}\\
xalan-2.7 & 99 & \textcolor[named]{Gray}{\rule{32mm}{\wwidth}}\\
ant-1.6 & 100 & \textcolor[named]{Gray}{\rule{33mm}{\wwidth}}\\
jedit-3.2 & 100 & \textcolor[named]{Gray}{\rule{33mm}{\wwidth}}\\
jedit-4.0 & 100 & \textcolor[named]{Gray}{\rule{33mm}{\wwidth}}\\
jedit-4.1 & 100 & \textcolor[named]{Gray}{\rule{33mm}{\wwidth}}\\
jedit-4.2 & 100 & \textcolor[named]{Gray}{\rule{33mm}{\wwidth}}\\
lucene-2.2 & 100 & \textcolor[named]{Gray}{\rule{33mm}{\wwidth}}\\
velocity-1.4 & 100 & \textcolor[named]{Gray}{\rule{33mm}{\wwidth}}
\end{tabular}
\end{center}
\caption{The median $\frac{learned}{optimal}\%$ result is 96\%
(at poi-2.5).}\label{fig:pm}
\end{figure}


\begin{figure*}[!t]
{
\begin{center}
\begin{tabular}{r|rrrrrrrr}
&  &  &      &   &  &   & Clustering \\
& &     OutDegree  &  Getters    &   Setters &NoM &InDegree  & Coefficient \\
data set&  $\beta_0$  &   $\beta_1$   &   $\beta_2$  &  $\beta_3$    &   $\beta_4$ &$\beta_5$ &$\beta_6$  \\\hline

ant-1.3  &-0.1351&0.0406&&-0.0183&&&&  \\
ant-1.4  &0.1459&0.0167&&0.0218&-0.0133&&&  \\
ant-1.5  &-0.0631&0.0135&-0.0325&0.0293&&&&  \\
ant-1.6  &-0.043&0.0466&-0.0454&-0.02&0.0486&&&  \\
ant-1.7  &-0.0564&0.046&-0.0562&-0.0283&0.0546&&&  \\
ivy-1.4  &-0.0066&0.0044&-0.0355&&0.0115&&&  \\
ivy-2.0  &-0.0386&0.013&&-0.0226&0.0087&&&  \\
jedit-3.2  &-1.0808&0.1099&0.4024&0.6921&&0.0522&&  \\
jedit-4.0  &-0.631&0.0441&0.3699&&&0.0532&&  \\
jedit-4.1  &-0.5152&0.057&0.1562&&&0.0378&&  \\
jedit-4.2  &-0.2777&0.0187&0.1275&&&0.0195&&  \\
jedit-4.3  &-0.0052&0.0016&0.0143&&-0.0007&&&  \\
lucene-2.0  &-0.7125&0.1277&&0.5153&0.0907&0.06&&  \\
lucene-2.2  &0.5616&0.0553&&1.0931&0.1688&&&  \\
lucene-2.4  &0.5217&0.0392&0.1497&-0.5903&0.1572&0.0675&&  \\
poi-1.5  &0.261&0.0593&0.1426&-0.3501&0.1113&&-1.5802&  \\
poi-2.0  &0.006&0.0047&&-0.0168&0.0087&&&  \\
poi-2.5  &-0.014&0.0331&0.1165&-0.1283&0.0387&&&  \\
poi-3.0  &-0.1698&0.0528&0.1648&-0.1918&0.0537&&&  \\
synapse-1.0  &0.025&0.022&&&&&&  \\
synapse-1.1  &-0.0713&0.0515&&0.0951&-0.0279&&&  \\
synapse-1.2  &-0.1312&0.0565&0.0771&&-0.0207&0.0154&&  \\
velocity-1.4  &0.8674&0.0198&0.0741&&-0.0104&0.0131&&  \\
velocity-1.5  &0.2875&0.1333&0.3349&-0.1941&&0.0258&0.8271&  \\
velocity-1.6  &0.2705&0.0814&0.1209&-0.218&0.0358&&0.9503&  \\
xalan-2.4  &-0.0722&0.0267&&&0.0075&&&  \\
xalan-2.5  &0.208&0.0301&0.0242&0.0426&&&-0.3072&  \\
xalan-2.6  &0.4958&0.0175&-0.0403&0.0707&0.02&&-0.5815&  \\
xalan-2.7  &1.2681&0.0155&-0.0453&0.0963&0.0042&&-0.4668&  \\
xerces-1.0  &0.1916&&0.2417&&0.0278&&&  \\
xerces-1.2  &-0.2139&0.0204&0.0298&-0.0397&&0.0129&0.6292&  \\
xerces-1.3  &-0.2716&0.1534&-0.0646&&&&&  \\
xerces-1.4  &-0.1181&0.141&0.2912&-0.4426&0.0865&0.0478&&  \\\hline
min&-1.0808&0.0016&-0.0646&-0.5903&-0.0279&0.0129&-1.5802&\\
median&-0.0386&0.0399&0.0968&-0.01915&0.0239&0.0378&-0.3072\\
max&1.2681&0.1534&0.4024&1.0931&0.1688&0.0675&0.9503\\\hline:
\%used&100\%&97\%&73\%&67\%&67\%&33\%&21\%&




\end{tabular}
\end{center}}
\caption{LSR results for 
$defects = \beta_0 + \sum^6_i\beta_ix_i$.
Sorted by the $\frac{learning}{optimal}$ ratio
seen for each data set.}\label{fig:eq}
\end{figure*}
\subsection{What Quasi-Social Metrics are ``Best''?}
\fig{eq} shows the co-efficients on the
linear regression
equations learned over the quasi-social metrics. The particular
linear regression tool we used (from the WEKA~\cite{witten05}) includes
a greedy back-select that removes those features that do not degrade
the overall performance of the equation.
Observe the blank cells of \fig{eq}:
\bi
\item
Different data sets selected from different
quasi-social metrics.
\item
No quasi-social metric was used in all data sets.
\ei
Also, the 
range of coefficients on the quasi-social metrics was very large
(observe the $min$, $max$ rows at the bottom of \fig{eq}.

Combining the results of
\fig{pm} and \fig{eq}, we say that:
\bi
\item
Some combination of quasi-social metrics was very useful in nearly all
the  data sets
studied here.
\item
The exact effect of particular metrics varied from data set to data set.
\ei
Hence, we do not recommend focusing on particular quasi-social metrics. Rather,
we recommend applying them all.

\section{Discussion}

\subsection{very good}

very good. compare to ~\cite{me10a}, must better.
\subsection{very simple}
note the absence of complex machinery: SVM, feature selection, random forests, etc.

Instead, a simple theory of human behavior (usage predicts for defects) plus a simple
data miner (linear regression) lead to predictions of very high competency (median results,
in \fig{pm}, within 3\% of optimal). Interesting... perhaps there has been too much
AI and not enough human intelligence in prior research on defect prediction.
\subsection{ANT}
Worst results in ANT. Why?
\section{Future Work}
Could apply the \cite{me10a} tools to here. But given the minimal room for improvement
(usually, less than 3\%), why bother?

More defect data sets.


%ACKNOWLEDGMENTS are optional
\section{Validity}
\section{Conclusion}
\section{Acknowledgments}
This work was conducted at Swinburne University and West Virginia
University with partial funding from the New Zealand Foundation for
Research, Science and Technology and the United States
National Science Foundation, CISE grant 71608561. 

% The following two commands are all you need in the
% initial runs of your .tex file to
% produce the bibliography for the citations in your paper.
\bibliographystyle{abbrv}
\bibliography{oldrefs,timrefs,rajrefs,markusrefs}  % sigproc.bib is the name of the Bibliography in this case
% You must have a proper ".bib" file
%  and remember to run:
% latex bibtex latex latex
% to resolve all references
%
% ACM needs 'a single self-contained file'!
%
%APPENDICES are optional
%\balancecolumns
%Appendix A
%\balancecolumns % GM June 2007
% That's all folks!
\end{document}
