
\subsection{Data}

%\subsection{Input Data Set}
%The field of software engineering has made a number of advances
%over the past few decades. However, only a small amount of serious
%effort has been directed towards creating and making available data
%sets that allow researchers to easily conduct new experiments and
%more importantly replicate other experiments. The Helix data set
%\cite{Helix10a} fills this specific gap along with similar efforts
%by Tempero et. al. \cite{Tempero10a} and the Promise data set
%currently maintained by Menzies et. al \cite{me07c}.
\begin{figure}
\begin{center}
\begin{tabular}{|r|l|}\hline
ant	& build Management system\\
ivy & build management system\\
jedit & text editor\\
lucene & text search engine\\
poi & API for Microsoft documents \\
synapse & an engneering service bus for Apache\\
velocity & template engine\\
xalan & XSLT processor\\
xerces & XML processor\\\hline
\end{tabular}
\end{center}
\caption{JAVA systems used in this study.}\label{fig:projects}
\end{figure}

The data used in this studies was built as a join between two on-line repositories:
\bi
\item The PROMISE repositoy~\cite{me07c} contains defect information for various open source OO systems at http://goo.gl/mStAY.
This defect data was collected by Jureczko \cite{Jureczko10a}.
 \item The  Helix repository at contains static source code data at http://goo.gl/nzXsu.
\ei
The joined data sets contains
33 releases of various ``long term'' projects; i.e. with
\emph{at least
15 releases} spanning over a development period of \emph{at least
36 months}. 

These projects, listed in \fig{projects},
fall into
three groups:
(a) \emph{applications}, (b) \emph{frameworks}, and (c) \emph{libraries}.
For a detailed description of these data sets,
see
\cite{Vasa10a}.
\begin{figure}[!t]
\begin{center}
\includegraphics[width=\columnwidth]{img/LinesOfCodeFit.pdf}
\end{center}
\caption{Lines of code (LOC) extracted from Bytecode is a very strong approximation of the LOC extracted directly from source code}
\label{fig:locfit}
\end{figure}


The set of metrics that we collect and the rationale for selecting
these measures in our study were shown in Figure \ref{fig:metrics}.
For LOC, we used an estimator based on the size of the compiled bytecode\footnote{How
was that done, exactly?}. As shown in \fig{locfit},
this estimator correlates very well with actual source code LOC.

The data extraction approach that 
we take for our analysis is illustrated in Figure \ref{fig:dataset}.
we need text showing how we
\footnote{not enough detail.
how to recognize setters?
how to  recognize getters
htow to 
recognize that class A depends on class B (this is needed for "InDegrees")
how is transitiveIndegree is computed
please add those notes to Appendix.tex or to here}

\begin{figure}[!t]
\begin{center}
\includegraphics[width=\columnwidth]{img/DataCollection.pdf}
\end{center}
\caption{We collect defect information from source control logs, while the metrics are collected from the compiled binaries}
\label{fig:dataset}
\end{figure}


\fig{defects} shows the distribution of defects seen in our classes.
This figure is sorted by the middle column (the median defects per class).
Usually, most classes have no defects;
but in 10\% of cases,
each class has more than 1 to 5 recorded defects.

\begin{figure}
\renewcommand{\baselinestretch}{0.7}
\small
\begin{center}
\begin{tabular}{l|r|rrrrr}
    &              &\multicolumn{5}{c}{percentiles}\\%\cline{3-7}
	data&\# classes&10\%&30\%& 50\%& 70\% &90\%\\\hline
ant-1.3 & 125 &   0.0 &   0.0 &   0.0 &   0.0 &   1.0\\
ant-1.4 & 178&    0.0 &   0.0 &   0.0 &   0.0 &   1.0\\
ant-1.5 & 293&   0.0 &   0.0 &   0.0 &   0.0 &   1.0\\
ant-1.6 & 351&   0.0 &   0.0 &   0.0 &   0.0 &   2.0\\
ant-1.7 & 493&    0.0 &   0.0 &   0.0 &   0.0 &   2.0\\
ivy-1.4 & 241&   0.0 &   0.0 &   0.0 &   0.0 &   0.0\\
ivy-2.0 &352&    0.0 &   0.0 &   0.0 &   0.0 &   1.0\\
jedit-3.2 & 272&   0.0 &   0.0 &   0.0 &   1.0 &   4.0\\
jedit-4.0 & 306&   0.0 &   0.0 &   0.0 &   0.0 &   2.0\\
jedit-4.1 & 312&   0.0 &   0.0 &   0.0 &   0.0 &   2.0\\
jedit-4.2 & 367&   0.0 &   0.0 &   0.0 &   0.0 &   1.0\\
jedit-4.3 & 492&   0.0 &   0.0 &   0.0 &   0.0 &   0.0\\
lucene-2.0 & 195&   0.0 &   0.0 &   0.0 &   1.0 &   4.0\\
poi-2.0 & 314&   0.0 &   0.0 &   0.0 &   0.0 &   1.0\\
synapse-1.0 & 157&    0.0 &   0.0 &   0.0 &   0.0 &   0.0\\
synapse-1.1 & 222&   0.0 &   0.0 &   0.0 &   0.0 &   1.0\\
synapse-1.2 & 256&   0.0 &   0.0 &   0.0 &   1.0 &   2.0\\
velocity-1.6 & 229&   0.0 &   0.0 &   0.0 &   1.0 &   2.0\\
xalan-2.4 & 428&   0.0 &   0.0 &   0.0 &   0.0 &   1.0\\
xalan-2.5 &763&    0.0 &   0.0 &   0.0 &   1.0 &   2.0\\
xalan-2.6 &875&    0.0 &   0.0 &   0.0 &   1.0 &   2.0\\
xerces-1.0 & 162&   0.0 &   0.0 &   0.0 &   2.0 &   2.0\\
xerces-1.2 &438&    0.0 &   0.0 &   0.0 &   0.0 &   1.0\\
xerces-1.3 &452&    0.0 &   0.0 &   0.0 &   0.0 &   1.0\\
lucene-2.2 & 247&   0.0 &   0.0 &   1.0 &   2.0 &   4.0\\
lucene-2.4 & 428&   0.0 &   0.0 &   1.0 &   2.0 &   5.0\\
poi-1.5 &237&    0.0 &   0.0 &   1.0 &   1.0 &   4.0\\
poi-2.5 &384&    0.0 &   0.0 &   1.0 &   2.0 &   2.0\\
poi-3.0 & 442&   0.0 &   0.0 &   1.0 &   1.0 &   2.0\\
velocity-1.4 & 196&    0.0 &   1.0 &   1.0 &   1.0 &   2.0\\
velocity-1.5 & 214&   0.0 &   0.0 &   1.0 &   2.0 &   4.0\\
xalan-2.7 & 908&   1.0 &   1.0 &   1.0 &   1.0 &   2.0\\
xerces-1.4 & 329&   0.0 &   0.0 &   1.0 &   2.0 &   4.0\\
\end{tabular}
\end{center}
\caption{Percentile distributions of number of defects per class. For example: in xalan-2.7, the median
number (50th percentile) of defects per class is 1; in lucene-2.4, 10\% of classes have 5 defects
or more; in xalan-2.7, most classes have at least 1 defect.
This table is sorted by the median defect count (see the 50\% percentile column).
}\label{fig:defects}
\end{figure}


\subsection{Experimental Rig}
This study used a
{\em (N=10)-way
cross-evaluation study}~\cite{witten05}.
The data set is divided
into $N=10$ buckets. For each bucket in the N-way, a predictor is
learned on the nine of the buckets, then tested on the remaining bucket.
These $N$ studies implements $N$ {\em hold out} study where
a model is tested on data {\em not} used in training.

To appreciate cross-validation, consider another approach called
{\em self-test} where the learned  model
is assessed on the {\em same} data that was used to create it.
Self-tests are deprecated by the research community~\cite{witten05}.
If the goal is to understand how 
well a defect predictor will work on future projects, 
it is best to assess the predictor via holdout modules 
not used in the generation of that predictor. 

In the WEKA 3.7.3 implementation of the cross-val used in this study, results are reported once for each test-instance
as that instance appears in one of the $N$ hold-outs\footnote{Note that prior to WEKA 3.7.2, the cross-val
procedure\newline{\tt java -cp weka.jar \$learner -t file.arff}\\ incorrectly return {\em self-test} results.}
So a data set containing $C$ examples will generated $C$ predictions,
regardless of the value of $N$ used for the number of hold-outs.

\subsection{Selection of Learners}
As mentioned above, there are very many methods for converting
static code measures into defect
predictors~\cite{mccabe76,lessmann09,taghi05a,taghi05,taghi00a,Harrison98a,
Briand99a,Briand00a,Nagappan06a,Menzies07a,Turhan09a,me10d}. We
adopted Holte's {\em simplicity-first} heuristic~\cite{holte93} and
applied a simple linear regression (LSR) algorithm from WEKA~\cite{witten05}, with no pre-preocessing.

Note that
WEKA's LSR tool uses a simple greedy back-select which is applied after the linear model is generated.
In that back-select, WEKA
steps through all the attributes removing the one with the smallest standardized coeffient until 
no improvement is observed in the estimate of the model error given by the Akaike information criterion. Hence, some 
attributes may be absent from the final learned model. 


Initally, we had plans to test various learners and feature extractors and instance selectors
and descritization methods (such as those we have used in the past~\cite{Menzies07a,Turhan09a,me10d})
but our results were so encouraging (see below) that there was little room for further improvement over simple LSR.





%Techniques that are commonly used for building models: Logistic
%regression (univariate and multivariate), Machine learning models.
%Area under Curve (AUC) used for identifying strength of the model.

%It is impractical to build a general purpose cross-project defect
%model \cite{Zimmermann09a, Jureczko10a, Weyeuker08a}

%Key points to get across:
%* Oracle suggests the inspection order -- very practical + easy to apply.
%* Defects may not be located in larger blocks of code.
%* Heavily used parts tend to change more, but do not have defects (across projects) -- probably an item to include in the discussion.

%\cite{Li93a} Li and Henry, JSS: 2 Ada projects. Show that CK metrics can be used as early indicators of classes that are most likely to undergo maintenance.
%
%\cite{Ohlsson96a} Olhsson, TSE: Predicting fault-prone software modules in telephone switches. Percentage modules Vs percentage faults investigation approach first shown.
%
%\cite{Basili96a} Basili, TSE: Validation of object-oriented metrics as quality indicators. 8 small student projects, minimal conceptual complexity. Show that size and coupling metrics (WMC, DIT, RFC, NOC, LCOM and CBO) are significant predictors, while cohesion was not a significant attribute.
%
% \cite{Harrison98a} Harrison, Journal or Emp. SE: Size based regression model to predict defects.
%
% \cite{Briand99a} Briand and Basili, TSE: 8 C++ student projects, Faults as detected during development. Argue that some CK metrics can be used as early indicators to identify fault-prone classes. Univariate Logistic regression. Some significance: WMC, DIT, RFC, NOC and CBO. Cohesion has no significance.
%
% \cite{Graves00a}, TSE: Use change logs to estimate distribution of fault incidences from change logs.
%
% \cite{Briand00a} Briand, JSS: Show that OO-metrics can be used to identify fault-prone classes.
%
% \cite{Fenton00a} Fenton, TSE: Did not observe any relationship between size and defect density. They argue that we can draw a broad conclusion that defect count would generally increase with size. However, a consistent and strong linear relationship between size and defect count is not always guaranteed.
%
% \cite{ElEmam01a} El Emam, JSS: Prediction of faulty classes using object oriented design metrics. 1 Java system. DIT and IDC have strong influence on fault-proneness. Small system (69 classes).
%
% \cite{ElEmam02a} El Emam, TSE: 1 large C++ system. Based on a statistical analysis of confounding effects of class size argues that none of the CK metrics are associated with fault-proneness. Hence they suggest that in studies that use OO metrics we must statistically control for size since size is often correlated with the metrics.
%
% \cite{Subramanyam03a} Subramanyam, TSE: 1 system (Java and C++ code base). WMC, CBO and DIT metrics associated with error counts.
%
% \cite{Succi03a} Succi, JSS: Using oo design metrics to detect defect-prone classes.
%
% \cite{Gyimothy05a} Gyimothy, TSE: Mozilla browser, 1 release. Show that CK metrics (except for NOC) are significantly associated with error proneness. Metrics recommended -- WMC, CBO, RFC, and SLOC.
%
% \cite{Ostrand05a} Ostrand, TSE: 2 large commercial systems. Evolution history analysed. Negative binomial regression model. Defect model is build from the history of a project -- same technique and method. but each project uses different parameters. This experiment has been extended to 4 systems with similar conclusions by Weyuker (2008, ESE 13 5 Oct).
%
% \cite{Hassan05a} Hassan and Holt, ICSM: Recently modified and corrected entities are more defect-prone.
%
% \cite{Zhou06a} Zhou, TSE: Using oo metrics for predicting high and low severity faults. NASA data set. Find that most oo metrics are statistically related to fault-proneness of classes across fault-severity. Metrics are able to better predict low severity faults better than high severity faults.
%
% \cite{Nagappan06a} Nagappan, ICSE: 5 commercial projects. No single set of metrics work for all projects. Defect prediction models may be accurate when obtained from similar projects (similar is not strongly defined).
%
% \cite{Pai07a} Pai, TSE: Logistic regression combined with bayesian network model. NASA data set. Bulk of the faults are found in a small set of modules. Reach similar conclusions as Zhou 2006. Also show that there is a relationship between CK metrics and faults.
%
% \cite{Li07a} Li and Shatnawi, JSS: Eclipse IDE, 3 releases. Multivariate logistic regression. Study between bad smells and class error proneness (including severity of errors). Argue that Shortgun surgery, God class and god methods are bad smells that should be refactored out as they are positively associated with defects within a class. Bad smells are detected using thresholds for metrics collected for each class.
%
% \cite{Koru07a} Koru, ICSE: There is no conclusive linear relationship between size and defects, although some argument can be made that there is a loose positive association between size and defects only because there is more code and hence there is a greater probability of defects. Found that there was a significant relationship between the logarithmic transformation of size and defect-proneness
%
% \cite{Kim07a} ICSE: Defects occur in bursts and past defects are useful in predicting next defect-prone areas. 7 open-source systems used to validate findings.
%
% \cite{Menzies07a} Menzies, TSE: The method used to build predictors is more important than the actual set of variables used to build the prediction model. Data mining static code attributes to learn defect predictors is useful. They note that if static code attributes capture little about code (as argued by Fenton, Shepherd), then we would expect much lower probabilities of detection and much higher false alarm rates. Also reject the use of simple thresholds for defect prediction. Probabilistic indicators, rather than categorical. 
%
% \cite{Moser08a} Moser, ICSE: 1 project. Eclipse. Change and Process metrics (code churn, past defects, refactoring applied, number of authors, file size and age) used to predict the presence (as well as absence) of defects.
%
% \cite{Aggarwal09a} 12 small Java systems. UG student work. Defects found during development. Coupling metrics are indicators of fault-proneness, inheritance metrics not so much.
%
% \cite{Zimmermann09a} Zimmermann, ESEC: Extension of work from Nagappan 2006 -- cross project defect prediction. Used 12 real-world projects. Found that each project is unique and the same model cannot be applied across projects. 
%
% \cite{Turhan09a} Turhan and Menzies, ESE: 10 software projects. Cross project defect prediction model. No single set of static code metrics will be applicable as defect predictors across projects.
%
% \cite{Hassan09a} Hassan, ICSE: Introduced information theoretic concept of entropy to measuring change. Evaluated 6 open-source systems and found that this approach was able to predict defects with a reasonable significance.
%
%[2010] Menzies, ASE:
%[2006] Janes, Information Sciences: Telecommunications system. Identification of defect-prone classes using design metrics.
%[2007] Olague, TSE: object-metrics can predict for faults. 1 project.
%[2008] Watanabe, Workshop on Predictive Models in SE: Cross validation across Java and C++ projects -- both text editors. Used prediction model build on the Java project to identify defects in C++ projects and vice-versa. Argue that is domain is similar and projects are similar size then the defect model may be used. Interesting idea, but limited by the size of the input data set.
% \cite{Cartwright00a} Cartwright, TSE: 1 system.
%
%
%
%
%

%\subsection{Aspects that need to be considered}
%This section contains points that need to be addressed in the paper:
%\begin{itemize}
%\item We use defect count in our study rather than defect density since less than 20\% (TODO: need precise number here) of the classes contain defects and approximately 5\% of the classes contain more than 3 defects.
%\item Defect count variability. Does having such a large set of zero values impact the models?
%\item Show that collinearity is not an issue with the variables that have been selected.
%\item 
%\end{itemize}
%
%
%% table of 
%
%\begin{figure*}
%\begin{center}
%\begin{tabular}{|c|c|}\hline
%a
%\\\hline
%\end{tabular}
%\end{center}
%\caption{Data used in this study.}\label{fig:data}
%\end{figure*}
