% This is "sig-alternate.tex" V1.8 June 2007
% This file should be compiled with V2.3 of "sig-alternate.cls" June 2007
%
% This example file demonstrates the use of the 'sig-alternate.cls'
% V2.3 LaTeX2e document class file. It is for those submitting
% articles to ACM Conference Proceedings WHO DO NOT WISH TO
% STRICTLY ADHERE TO THE SIGS (PUBS-BOARD-ENDORSED) STYLE.
% The 'sig-alternate.cls' file will produce a similar-looking,
% albeit, 'tighter' paper resulting in, invariably, fewer pages.
%
% ----------------------------------------------------------------------------------------------------------------
% This .tex file (and associated .cls V2.3) produces:
%       1) The Permission Statement
%       2) The Conference (location) Info information
%       3) The Copyright Line with ACM data
%       4) NO page numbers
%
% as against the acm_proc_article-sp.cls file which
% DOES NOT produce 1) thru' 3) above.
%
% Using 'sig-alternate.cls' you have control, however, from within
% the source .tex file, over both the CopyrightYear
% (defaulted to 200X) and the ACM Copyright Data
% (defaulted to X-XXXXX-XX-X/XX/XX).
% e.g.
% \CopyrightYear{2007} will cause 2007 to appear in the copyright line.
% \crdata{0-12345-67-8/90/12} will cause 0-12345-67-8/90/12 to appear in the copyright line.
%
% ---------------------------------------------------------------------------------------------------------------
% This .tex source is an example which *does* use
% the .bib file (from which the .bbl file % is produced).
% REMEMBER HOWEVER: After having produced the .bbl file,
% and prior to final submission, you *NEED* to 'insert'
% your .bbl file into your source .tex file so as to provide
% ONE 'self-contained' source file.
%
% ================= IF YOU HAVE QUESTIONS =======================
% Questions regarding the SIGS styles, SIGS policies and
% procedures, Conferences etc. should be sent to
% Adrienne Griscti (griscti@acm.org)
%
% Technical questions _only_ to
% Gerald Murray (murray@acm.org)
% ===============================================================
%
% For tracking purposes - this is V1.8 - June 2007


\documentclass{sig-alternate}
\usepackage{multirow}
\usepackage{multicol}
\usepackage{graphicx}
\usepackage{url}
\usepackage{times}

\newenvironment{smallitem}{
   \setlength{\topsep}{0pt}
   \setlength{\partopsep}{0pt}
   \setlength{\parskip}{0pt}
   \begin{itemize}
   \setlength{\leftmargin}{.2in}
   \setlength{\parsep}{0pt}
   \setlength{\parskip}{0pt}
   \setlength{\itemsep}{0pt}}{\end{itemize}}

\newenvironment{smallnums}{
   \setlength{\topsep}{0pt}
   \setlength{\partopsep}{0pt}
   \setlength{\parskip}{0pt}
   \begin{enumerate}
   \setlength{\leftmargin}{.2in}
   \setlength{\parsep}{0pt}
   \setlength{\parskip}{0pt} \setlength{\itemsep}{0pt}}{\end{enumerate}}

\newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
\newcommand{\be}{\begin{enumerate}}
\newcommand{\ee}{\end{enumerate}}



%\usepackage{epsf}
% In order to use the figure-defining commands in ieeefig.sty...
%\usepackage{ieeefig}
%\usepackage{latex8}

\begin{document}




%
% --- Author Metadata here ---
\conferenceinfo{ICSE PROMISE Workshop}{2008 Leipzig, Germany}
\CopyrightYear{2008} % Allows default copyright year (200X) to be over-ridden - IF NEED BE.
%\crdata{0-12345-67-8/90/01}  % Allows default copyright data (0-89791-88-6/97/05) to be over-ridden - IF NEED BE.
% --- End of Author Metadata ---

\title{Comparing Design and Code Metrics for Software Quality Prediction}
%\title{Alternate {\ttlit ACM} SIG Proceedings Paper in LaTeX
%Format\titlenote{(Produces the permission block, and
%copyright information). For use with
%SIG-ALTERNATE.CLS. Supported by ACM.}}
%\subtitle{[Extended Abstract]
%\titlenote{A full version of this paper is available as
%\textit{Author's Guide to Preparing ACM SIG Proceedings Using
%\LaTeX$2_\epsilon$\ and BibTeX} at
%\texttt{www.acm.org/eaddress.htm}}}
%
% You need the command \numberofauthors to handle the 'placement
% and alignment' of the authors beneath the title.
%
% For aesthetic reasons, we recommend 'three authors at a time'
% i.e. three 'name/affiliation blocks' be placed beneath the title.
%
% NOTE: You are NOT restricted in how many 'rows' of
% "name/affiliations" may appear. We just ask that you restrict
% the number of 'columns' to three.
%
% Because of the available 'opening page real-estate'
% we ask you to refrain from putting more than six authors
% (two rows with three columns) beneath the article title.
% More than six makes the first-page appear very cluttered indeed.
%
% Use the \alignauthor commands to handle the names
% and affiliations for an 'aesthetic maximum' of six authors.
% Add names, affiliations, addresses for
% the seventh etc. author(s) as the argument for the
% \additionalauthors command.
% These 'additional authors' will be output/set for you
% without further effort on your part as the last section in
% the body of your article BEFORE References or any Appendices.

\numberofauthors{1} %  in this sample file, there are a *total*
% of EIGHT authors. SIX appear on the 'first-page' (for formatting
% reasons) and the remaining two appear in the \additionalauthors section.
%
\author{
% You can go ahead and credit any number of authors here,
% e.g. one 'row of three' or two rows (consisting of one row of three
% and a second row of one, two or three).
%
% The command \alignauthor (no curly braces needed) should
% precede each author name, affiliation/snail-mail address and
% e-mail address. Additionally, tag each line of
% affiliation/address with \affaddr, and tag the
% e-mail address with \email.
%
% 1st. author
\alignauthor
Bojan Cukic, Yue Jiang, Tim Menzies, Nick Bartlow\\
       \affaddr{The Lane Department of Computer Science and Electrical Engineering}\\
       \affaddr{West Virginia University}\\
       \affaddr{Morgantown, WV26505}\\
       \email{\{cukic, yue, bartlow\}@csee.wvu.edu; tim@menzies.us}
% 2nd. author
%\alignauthor
%G.K.M. Tobin\titlenote{The secretary disavows
%any knowledge of this author's actions.}\\
%       \affaddr{Institute for Clarity in Documentation}\\
%       \affaddr{P.O. Box 1212}\\
%       \affaddr{Dublin, Ohio 43017-6221}\\
%       \email{webmaster@marysville-ohio.com}
% 3rd. author
%\alignauthor Lars Th{\o}rv{\"a}ld\titlenote{This author is the
%one who did all the really hard work.}\\
%       \affaddr{The Th{\o}rv{\"a}ld Group}\\
%       \affaddr{1 Th{\o}rv{\"a}ld Circle}\\
%       \affaddr{Hekla, Iceland}\\
%       \email{larst@affiliation.org}
%\and  % use '\and' if you need 'another row' of author names
% 4th. author
%\alignauthor Lawrence P. Leipuner\\
%       \affaddr{Brookhaven Laboratories}\\
%       \affaddr{Brookhaven National Lab}\\
%       \affaddr{P.O. Box 5000}\\
%       \email{lleipuner@researchlabs.org}
% 5th. author
%\alignauthor Sean Fogarty\\
%       \affaddr{NASA Ames Research Center}\\
%       \affaddr{Moffett Field}\\
%       \affaddr{California 94035}\\
%       \email{fogartys@amesres.org}
% 6th. author
%\alignauthor Charles Palmer\\
%       \affaddr{Palmer Research Laboratories}\\
%       \affaddr{8600 Datapoint Drive}\\
%       \affaddr{San Antonio, Texas 78229}\\
%       \email{cpalmer@prl.com}
%}
% There's nothing stopping you putting the seventh, eighth, etc.
% author on the opening page (as the 'third row') but we ask,
% for aesthetic reasons that you place these 'additional authors'
% in the \additional authors block, viz.
%\additionalauthors{Additional authors: John Smith (The Th{\o}rv{\"a}ld Group,
%email: {\texttt{jsmith@affiliation.org}}) and Julius P.~Kumquat
%(The Kumquat Consortium, email: {\texttt{jpkumquat@consortium.net}}).}
%\date{30 July 1999}
% Just remember to make sure that the TOTAL number of authors
% is the number that will appear on the first page PLUS the
% number that will appear in the \additionalauthors section.
}
\maketitle
\begin{abstract}

The prediction of fault-prone modules continues to attract interest due to the significant impact it has on software quality assurance. One of the most important goals of such techniques is to accurately predict the modules where faults are likely to hide as early as possible in the development lifecycle.  Design, code, and most recently, requirements metrics have been successfully used for predicting fault-prone modules.  The goal of this paper is to compare the performance of predictive models which use design-level metrics with those that use code-level metrics.  We analyze thirteen datasets from NASA Metrics Data Program which offer design as well as code metrics. Using a range of modeling techniques and statistical significance tests, we confirmed that models built from code metrics typically outperform design metrics based models.  However, both types of models prove to be useful as they can be constructed in different project phases. Code-based models can be used to increase the performance of design-level models and, thus, increase the efficiency of assigning verification and validation activities late in the development lifecycle.
\end{abstract}

% A category with the (minimum) three required fields
%\category{H.4}{Information Systems Applications}{Miscellaneous}
%A category including the fourth, optional field follows...
%\category{D.2.8}{Software Engineering}{Metrics}[complexity measures, performance measures]

%\terms{Delphi theory}

%\keywords{ACM proceedings, \LaTeX, text tagging}

\section{Introduction}
Elsewhere we have documented ceiling effects in the performance of defect
predictors~\cite{Menzies08}. Despite much work, our current generation of learners are not
finding new information in the PROMISE data sets. We have explained that
observation via `limited information hypothesis'; i.e. that the insights
that can be gained from static code measures are (1) easy to find and (2)
limited in number. If so, then the future research into defect prediction
should {\em not} focus on better algorithms. Rather, it should focus on
improving either (a)~the evaluation functions of those algorithms (e.g. to
inject more business knowledge) or (b)~the information content of the
training data.

This paper explores option (b). Previously, we have had
much success with augmenting static code measures with
features extracted from requirements documents via lightweight text parsing.
Figure~\ref{fig:Pc5ROCandPR} shows the results~\cite{Jiang07}:  note the
remarkable improvement when learning was applied
to features extracted from {\em both} code {\em and} requirements.

Previously, we have had much success with
augmenting static code measures with features extracted from requirements
documents via lightweight text parsing(referring to Figure~\ref{fig:Pc5ROCandPR} from~\cite{Jiang07}). Note the remarkable improvement when learning was applied to both sets of features.
\begin{figure}[h]
%\begin{center}
$\begin{array}{c@{\hspace{0.3in}}c}
%\multicolumn{1}{l}{}} &
%	\multicolumn{1}{l}{}} \\
%[-0.53cm]
\epsfxsize=1.5in
\epsffile{cm1_ij_mod_req.ps} &
	\epsfxsize=1.5in
	\epsffile{pc1_ij_mod_req.ps} \\ [0.4cm]
\mbox{\bf (a)} & \mbox{\bf (b)}
\end{array}$
%\end{center}
\caption{(ROC curves for (a) CM1 and (b) PC1. Results from
CM1 and PC1 after a 10-way cross validation.
The ideal spot on these ROC curves is top left; i.e. no false alarms
and perfect detection ($\{pd,pf\}=\{1,0\}$). The dashed lines on
those plots show $\{pd,pf\}$ results when fault prediction models
used features mined from requirements text,
or features mined from module measures (in isolation).
The solid lines
show the results of models which used these two kinds of features in
combination. From [17].}
\label{fig:Pc5ROCandPR}
\end{figure}
While those results were promising, those conclusions were based on a
limited sample size. Utilizing our NASA contacts, we could only find
requirements data for 3 of the defect data sets in PROMISE.  The same is
{\em not} true for design metrics- this paper reports results from the 13
PROMISE data sets where we could find design-level metrics.

In this paper, after describing our data,
we show results from learning defect predictors from
\bi
\item just static code attributes;
\item just design code attributes;
\item Or from a combination of both.
\ei
In a result consistent with our previous results, we find that combining
attributes yields better detectors than (1) or (2) in isolation. The
improvement is not as dramatic as before but the general thesis is endorsed:
defect detectors can be improved by increasing the information content of
the training set. We recommend therefore that, in future, researchers
explore the effects of conjunctions of attributes from multiple sources.

The remainder of the work is broken down as follows: Section 2 descibes the NASA datasets and metrics used in the study, Section 3 outlines the experimental design in terms of the chosen algorithms and selected methods of testing, Section 4 presents the experimental results and a discussion of considerations, Section 5 provides a background of related work, and Section 6 concludes with a summary and future work.

\section{Metrics Description}
\subsection{NASA MDP metrics}
The datasets used in this study come from the NASA Metrics Data Program
(MDP) data repository~\cite{mdp} that contains 13 projects shown in Table~\ref{tab:dataset}.

\begin{table*}
\centering
\small
\caption{Datasets used in this study}
\label{tab:dataset}
\begin{tabular}{|c|c|c|c|c|c|l|c|}
\hline \textbf{Data}&    \textbf{mod.}\#  & \% \textbf{faulty} &\multicolumn{3}{|c|}{\# \textbf{metrics}} & \textbf{note} & \textbf{lang.}\\
\cline{4-6}         & & & \textbf{all}& \textbf{design}&  \textbf{code}  &&\\
\hline CM1 &	505&		16.04\%& 40&16 &20 	&Spacecraft instrument&C\\
\hline KC1&	2407&		13.9\% & 21&4 & 17   &    storage management for receiving/processing ground data&C++\\
\hline KC3&	458&		6.3\%  & 40&16 & 20   &    Storage management for ground data&Java\\
\hline KC4&	125&		48\%   & 40&16 & 20   &    a ground-based subscription server&Perl\\
\hline PC1&	1107&		6.59\% & 40&16 & 20   &    flight software from an earth orbiting satellite&C\\
\hline PC3&	1563&		10.43\%& 40&16 & 20   &    Flight software for earth orbiting satellite&C\\
\hline PC4&	1458&		12.24\%& 40&16 & 20   &    Flight software for earth orbiting satellite&C\\
\hline MW1&	433&		6.7\%  & 40&16 & 20   &    a zero gravity experiment related to combustion	&C\\
\hline MC2&	161&		32.30\%& 40&16 & 20   &     a video guidance system&C++\\
\hline JM1 &    10,878&		19.3\% & 21&4 &	 17       & a real time predictive ground system&	C\\	
\hline MC1 &	9466&		0.64\% & 39&15 &20	        & a combustion experiment of a space shuttle &(C)C++\\
\hline PC2 &	5589&		0.42\% & 40&16 &20    &dynamic simulator for attitude control systems&	C\\
\hline PC5 &   17,186&		3.00\% & 39&15 &20     &a safety enhancement of a cockpit upgrade system &	C++\\
\hline
\end{tabular}
\end{table*}



The MDP repository provides module metrics
that describe the software artifacts from $13$ NASA projects. In MDP data, JM1 and KC1 have $24$ total attributes; MC1 and PC5 have $42$ total attributes; the remaining $9$ datasets
have $43$ modules metrics. All $13$ MDP datasets have one $module\_id$ attribute and two error-related attributes: $error\_count$ and $error\_density$. We remove the $module\_id$ attribute and the $error\_density$ attributes. The error\_count attribute is converted into a boolean attribute called $DEFECT$. If the $error\_count$ attribute is greater or equal $1$, than it's value is $TRUE$, otherwise it is $FALSE$. Thus, after removing and replacing these attributes, JM1 and KC1 have $21$ attributes that can be used as predictor variables, MC1 and PC5 have $39$, the other datasets have $40$.

%code metrics: jm1 and kc1 has 17 metrics, other data have 20 metrics

%kc1 and jm1 have 4 design metrics
%mc1 and pc5 have 15 design metrics
%other 9 datasets have 16 design metrics

%full: jm1 and kc1 have 21 metrics
%mc1 and pc5 have 39 all metrics
%other have 40 metrics

MDP module metrics shown in Table~\ref{tab:metric} are extracted by using McCabe IQ 7.1, a reverse engineering tool that is able to automatically extract software quality metrics, visualize flowgraphs and generate report documents~\cite{Mccabe}.
%Some of the formulae shown in Table~\ref{tab:metric} are based on the description provided in MDP data %repository~\cite{mdp}, some are based on the white papers provide by McCabe company~\cite{Mccabe}.

It is not unusual to derive design metrics from code in reverse engineering that has been studied by many works and there are many tools developed to do so~\cite{CanforaHarman07,Briand06,Antoniol01,Antoniol02,Systa00,Tonella05}. McCabe IQ 7.1 is one of reverse engineering tools that generates flowgraphs through analyzing source code and calculates design metrics based on flowgraphs. We divide the MDP module metrics into three groups: $design$, $code$, and $other$ metrics (Table~\ref{tab:metric}). The design metrics are the metrics that can be derived from flowgraphs, such as $node\_count$, $edge\_count$, and McCabe cyclomatic complexity related metrics. Our criteria to separate design metrics from module metrics is whether the metrics have the flexibility to be generated from design flowgraphs (diagrams) when a software system is in design phase. The code metrics are based on static code, such as $num\_operators$, $num\_operands$, and Halstead metrics calculated from number of operators and operands~\cite{fenton97}. The other metrics are related to design and code metrics (calculated by mathematical operation between design and code metrics).  Each of the datasets have 4 $other$ metrics with the exception of JM1 and KC1 that have $0$.  Additionally, we define a group of metrics of $all$ to include the entire set of metrics found in each of the three categories: $design$, $code$, and $other$.

\begin{table*}
\centering
\small
\caption{Metrics used in this study}
\label{tab:metric}
\begin{tabular}{|l|l|l|}
\hline group & metrics&	description or	formula \\	
\hline \multirow{20}{*}{code}& PARAMETER\_COUNT& 	  	Number of parameters to a given module	\\  	
& NUM\_OPERATORS:N1	&The number of operators contained in a module	  	\\
& NUM\_OPERANDS:N2  &	The number of operands contained in a module	  	\\
& NUM\_UNIQUE\_OPERATORS:$\mu_{1}$	& The number of unique operators contained in a module	\\  			
& NUM\_UNIQUE\_OPERANDS:$\mu_{2}$&	The number of unique operands contained in a module	  	\\
& HALSTEAD\_CONTENT:$\mu$&	The halstead length content of a module	  $\mu= \mu_{1}+ \mu_{2}$\\	
& HALSTEAD\_LENGTH:N&	The halstead length metric of a module	  $N=N_1+N_2$	\\
& HALSTEAD\_LEVEL:L &	The halstead level metric of a module	  $L= \frac{(2*\mu_{2})}{\mu_{1}*N_2}$\\	
& HALSTEAD\_DIFFICULTY:D &	The halstead difficulty metric of a module	  $D=\frac{1}{L}$\\			
& HALSTEAD\_VOLUME:V	& The halstead volume metric of a module	  $V=N*\log_2(\mu_{1}+ \mu_{2})$\\
& HALSTEAD\_EFFORT:E	&The halstead effort metric of a module	  $E=\frac{V}{L}$	\\
& HALSTEAD\_PROG\_TIME:  T &	The halstead programming time metric of a module 	  $T=\frac{E}{18}$\\	
& HALSTEAD\_ERROR\_EST:  B	& The halstead error estimate metric of a module	  $B=\frac{E^{2/3}}{1000}$\\
& NUMBER\_OF\_LINES 	 & 	Number of lines in a module	  	\\
& LOC\_BLANK 	&  	The number of blank lines in a module\\	  	
& LOC\_CODE\_AND\_COMMENT:NCSLOC 	&  	The number of lines which contain both code and comment in a module	\\	
& LOC\_COMMENTS 	&  	The number of lines of comments in a module	\\
& LOC\_EXECUTABLE 	 & 	The number of lines of executable code for a module (not blank or comment)\\
& PERCENT\_COMMENTS &	  	Percentage of the code that is comments	  	\\
& LOC\_TOTAL &	  	The total number of lines for a given module	 \\ 	
\hline \multicolumn{3}{|c|}{}\\
\hline \multirow{15}{*}{design}&EDGE\_COUNT:e &	Number of edges found in a given module control from one module to another \\
 & NODE\_COUNT:n  &	Number of nodes found in a given module \\
 & BRANCH\_COUNT  & Branch count metrics	  	\\
 & CALL\_PAIRS 	&  	Number of calls to other functions in a module	  	\\
 & CONDITION\_COUNT &	  	Number of conditionals in a given module	  	\\
 & CYCLOMATIC\_COMPLEXITY: v(G)&	The cyclomatic complexity of a module	  $v(G) = e - n +2$\\	
 & DECISION\_COUNT 	&  	Number of decision points in a given module	  	\\
 & DECISION\_DENSITY 	&  		  $Condition\_count / Decision\_count$	\\
 & DESIGN\_COMPLEXITY:iv(G)& 	The design complexity of a module	  	\\
 & DESIGN\_DENSITY 	&  	Design density is calculated as: iv(G)/v(G)	  $\frac{iv(G)}{v(G)}$\\	
 & ESSENTIAL\_COMPLEXITY:ev(G)&	The essential complexity of a module	\\  	
 & ESSENTIAL\_DENSITY 	  &	Essential density is calculated as: 	  $\frac{(ev(G)-1)}{(v(G)-1)}$\\	
 & MAINTENANCE\_SEVERITY 	& 	Maintenance Severity is calculated as:  $\frac{ev(G)}{v(G)}$\\	  	
 & MODIFIED\_CONDITION\_COUNT & The effect of a condition affect a decision outcome by varying that condition only\\
 & MULTIPLE\_CONDITION\_COUNT &	  	Number of multiple conditions that exist within a module	  	\\
 & PATHOLOGICAL\_COMPLEXITY  &	A measure of the degree to which a module contains extremely unstructured constructs	 \\
 \hline \multicolumn{3}{|c|}{}\\
\hline \multirow{4}{*}{others} & NORMALIZED\_CYLOMATIC\_COMPLEXITY 	&  		$\frac{v(G)}{NUMBER\_OF\_LINES}$\\	
 & GLOBAL\_DATA\_COMPLEXITY:gdv(G)&  	the ratio of cyclomatic complexity of a module's structure  to its parameter\_count \\
 & GLOBAL\_DATA\_DENSITY  & Global Data density is calculated as:  $\frac{gdv(G)}{v(G)}$\\	
 & CYCLOMATIC\_DENSITY  & 	 $\frac{v(G)}{NCSLOC}$\\	
\hline
\end{tabular}
\end{table*}


\section{Experimental Design}
%explain how the experiment are conducted.
%using learners in weka to run on the code, design, and all data set.
%crossvalidation to get roc
% to calculated AUC
% %using boxplot to show the mean and variance
%using nonparametric statistical tests to test.
We use five machine learners from Weka in this experiment~\cite{Witten05}. Recall that there are $13$ MDP datasets, each having three different groups of metrics: $design$, $code$, and $all$.  The predicted variable is whether a module contains one or more defects, that is, whether a module is fault-prone module or not. The Receiver Operating Characteristic (ROC) curve is used to measure the performance of different metrics.  In total, $1,950 $ ROC curves are generated \emph{(13 datasets * 3 metric groups * 5 machine learners * 10 runs)}.  Each ROC curve is generated using $10$ by $10$ way cross validation through the WEKA machine learning package~\cite{weka05}.  Additionally, the Area Under the Curve (AUC) is calculated for each of the $1,950$ ROC curves using the Trapezoid rule.  To visualize the results, a boxplot is used to show statistics for each of the $10$ AUCs from each dataset according to a single group of metrics and one of the five chosen machine learners.  We compare the performance of $design$ metrics, $code$ metrics, and $all$ metrics for a specific machine learner on each dataset. Since we use $5$ machine learners on three different groups of metrics over $13$ datasets, there are $5*13=65$ boxplot diagrams. Due to space limitations we are unable to show all boxplots, therefore we chose to only show the performance of the best learners on each group of metrics for each dataset.

In order to further investigate whether the performance of three different groups of metrics have any statistical significant difference, we use nonparametric statistical tests according to Demsar's recommendation~\cite{demsar06}. First, we use the Friedman test to test whether there is difference among the best learners of three groups of metrics over 13 MDP datasets. Then, we use Wilcoxon test to conduct pairwise tests on different groups in each dataset, that is, three tests are conducted in each dataset: (1) the performance of the best learner on metrics $all$ $vs.$ the performance of best learner on $code$ metrics; (2) the performance of the best learner on $all$ metrics $vs.$ that of $design$ metrics; and the performance of the best learner on $code$ metrics $vs.$ that of $design$ metrics.

In the remainder of this section, we first introduce the five machine learners we used, and then we discuss ROC curves and AUC. Next, we describe the boxplot diagram we used to compare the performance of different groups of metrics. Finally, we demonstrate our statistical tests and corresponding hypotheses we used in this experiment.

\subsection{Machine Learners}
We build predictive models using machine learners from Weka
package~\cite{weka05} as shown in Table~\ref{tab:learner}. All these
machine learning algorithms are used with their default parameters except random forest. In this experiment, random forest is built with 500 trees (default is 10 trees in Weka) because in the original algorithm of random forest proposed by Breidman~\cite{Breiman01}, the default number of trees is 500.

\begin{table}[h]
\centering
\caption{Machine learners used in this study}
\label{tab:learner}
\begin{tabular}{|c|l|l|}
\hline &	learner&	Abbrev.\\
\hline 1&	Random Forest&	rf\\
\hline 2&	Bagging&	bag\\
\hline 3&	Logistic regression&	lgi\\
\hline 4&	Boosting&	bst\\
\hline 5&	Naivebayes& 	nb\\
\hline
\end{tabular}
\end{table}

Random Forest (rf) is a decision tree-based classifier that is proved to have good performance by Lan $et$ $al$~\cite{guo04}. As implied from its name, it builds a ``forest" of decision trees. The trees are constructed using the following strategy: \bi \item{
The root node of each tree contains a bootstrap sample data of the same size as the original data. Each tree has a different bootstrap sample.} \item{At each node, a subset of variables are randomly selected from all the input variables to split the node and the best split is adopted.} \item{Each tree is grown to the largest extent possible
without pruning.} \item{When all trees in the forest are built, new instance(s) is fitted to all the trees and a voting process is taken place. The forest selects the classification with the most votes as the prediction of new instance(s).} \ei

NaiveBayes (nb)is based on Bayes's rule and ``naively" assumes data independence. This assumption may be considered overly simplistic in real life application scenarios. It is proposed to effectively predict fault-proneness of MDP dataset by~\cite{menzies07}.

Bagging (bag) stands for bootstrap aggregating which relies on an ensemble of different models. The training data is resampled based on one original dataset. It performs significantly better than the single model built from the original training data and it is never substantially worse according to Witten and Frank~\cite{Witten05}.

Boosting (bst) combines multiple models by explicitly seeking models that complement one another. First, it is similar to bagging in using voting for classification or averaging for numeric prediction. It combines models of the same type like bagging. However, boosting is iterative. ``Whereas in bagging individual models are built separately, in boosting each new model is influenced by the performance of those built previously. Boosting encourages new models to become experts for instances handled incorrectly by earlier ones."~\cite{Witten05}.

Logistic regression(lgi) is a classification scheme which uses a well defined mathematical logistic regression function.  The most popular models are generalized linear models. It makes use of several predictor variables that may be either numerical or discrete to predict the class.

\subsection{ROC Curves}
%show one ROC curve of design vs. code vs. all
% ten way crossvalidation
Receiver Operating Characteristic (ROC) curves provide an intuitive
way to compare the classification performance of different metrics. It is a plot
of the Probability of Detection ($PD$) as a function of the Probability of False alarm ($PF$) across all the possible experimental threshold settings. Many classification algorithms allow users to define and adjust a threshold parameter in order to generate an appropriate classifier~\cite{Witten05}.  When modeling software quality prediction a higher PD can be produced at the cost of higher PF and vice versa. A typical ROC curve has a concave shape with (0,0) as the beginning and (1,1) as the end point. Figure~\ref{fig:ROC} shows three example ROCs of $all$, $code$, and $design$ metrics over dataset PC5.

\begin{figure}[h]
\begin{center}
  % Requires \usepackage{graphicx}
  \includegraphics[scale=0.4]{pc5Roc.ps}
  \caption{ROC curve of PC5}\label{fig:ROC}
\end{center}
\end{figure}


The Area Under the ROC curve, referred to as AUC, is a numeric performance evaluation measure that is directly associated with an ROC curve. It is very common to use AUC to compare the performance of different datasets or classifiers. From Figure~\ref{fig:ROC}, we can see that the performances of the best learners on three different metrics are similar although the performance of random forest on $all$ metrics are slightly better than that of bagging on $code$ metrics, and the performance of $design$ metrics is the worst among three of them. The values of AUCs demonstrate the same trends; AUCs of $all$, $code$, and $design$ are 0.979, 0.967, and 0.956 respectively.  Here, $all$ (0.979) > $code$ (0.967) > $design$ (0.956) and the differences among the values is small: $all-code = 0.012$, $code-design = 0.011$, $all-design =0.023$.

We use 10 by 10 way cross-validation in this experiment. Cross-validation is the statistical practice of partitioning
sample of data into two subsets: training and testing subset.  The training subset trains the predictors
and the testing subset validates the predictors. $90\%$ is randomly assigned to the training subset and the remaining $10\%$ of data is used for testing. The data is randomly divided into 10 fixed bins with equal size. We leave one bin to act as test data, and the other 9 bins is used to train the learners.

\subsection{Boxplot Diagrams}
% show one dataset here
A boxplot, also known as a box and whisker diagram, graphically shows groups of a numerical data distribution using five first order statistics: the smallest observation, lower quartile (Q1), median, upper quartile (Q3) and the largest observation.  The box is constructed based on the interquartile range (IQR) from Q1 to Q3. The line inside the box depicts the median which follows central tendency. The whiskers indicate the smallest observation and the largest observation. Figure~\ref{fig:boxpc5} shows an example boxplot of the best learners of three groups of metrics on PC5 dataset: random forests in $all$ metrics has the best performance (the largest values of AUCs) and the performance of bagging in $design$ metrics has the worst of the three.

\begin{figure}[h]
\begin{center}
  % Requires \usepackage{graphicx}
  \includegraphics[scale=0.4]{pc5.ps}
  \caption{Boxplot of PC5}\label{fig:boxpc5}
\end{center}
\end{figure}

\subsection{Statistical Significance Tests}
The most popular method used to evaluate a classifier on a dataset is $10$ by $10$ ways cross-validation ($10x10$ CV).  The $10x10$ CV results in $10$ individual values of AUC. These $10$ values are usually similar to each other, given that they come from the same population after randomization.  With only $10$ values it is difficult to say whether the values follow the normal distribution (i.e., indicate that they obey the central limit theorem).  Therefore, using parametric statistical methods which assume a normally distributed population to compare the performance of classifiers may not be justified.  A prudent approach calls for the use of nonparametric methods.  The loss of efficiency caused by using nonparametric tests is typically marginal~\cite{conover99,siegel56}.

Demsar~\cite{demsar06} overviewed theoretical work on statistical tests for multiple classifiers over multiple datasets.  He recommended the Wilcoxon signed rank test for the comparison of two classifiers and the Friedman test with the corresponding post-hoc tests when the comparison includes more than two classifiers. Wilcoxon signed rank test and Friedman's test are nonparametric counterparts for paired $t-test$ and analysis of variance (ANOVA) parametric methods, respectively. Demsar advocates these tests largely due to the fact that nonparametric procedures make less stringent demands on the data. However, two issues need attention. First, nonparametric tests do not utilize all the information available. The actual data values (in our case, for example, AUCs) are not used in the test procedure. Instead, the signs or ranks of the observations are used. Therefore, nonparametric procedures will be more powerful then their parametric counterparts, when justifiably used.  The second point is that signed rank tests are constructed for the null hypothesis that the difference of the performance measure is symmetrically distributed. For non-symmetric distributions, this test can lead to a wrong conclusion.

All statistical tests conducted here utilize the routines provided in a statistical package R~\cite{R05}.
Based on Demsar's recommendation, we use the Friedman test to test whether there is difference in the performance among 3 groups of metrics over $13$ MDP datasets. The p-value from the Friedman test is $3.604e-05$ which is far less than $0.05$ (we use a $95\%$ confidence interval), which strongly suggests there is significant difference among different groups of metrics over 13 datasets.  Thus, we need pairwise comparisons of different groups of metrics on 13 datasets.

Assume we are looking at two different groups of metrcs, $A$ and $B$, and that we have their performance for a chosen learner on a single dataset.  In order to find out which set of metrics has a better performance than the other, the appropriate statistical test hypotheses are:
\\
\emph{$H_0$:  There is no difference in the performance of group of A and B metrics;}\\
\emph{$H_1$:  The performance of group of metrics A is better than that of B;}\\
\emph{$H_2$:  The performance of group of metrics A is worse than that of B.}\\

At first, we test whether two groups of metrics have the same performance on a specified learner in a given confidence interval (95\% confidence interval used in this study). If two learners do not have any significant difference in a given confidence interval (p-value greater than 0.05 in this case), then further tests on $H_1$ and $H_2$  are not necessary, thus, $H_0$ is the correct hypothesis: there is no difference in the performance of group $A$ and $B$ metrics. Otherwise, the further test of $H_1$ is conducted. If the outcome p-value of $H_1$ is less than $0.05$, then $H_1$ is accepted: the performance of group of metrics A is better than that of B. $H_2$ will be executed if $H_1$ is rejected. After conducting these three hypothesis tests, the relationship in the performance of group A and B metrics will be clear: $H_0$ (A=B), $H_1$ (A>B), and $H_2$ (A<B).
%In some rare situations, $H_0$ shows there is significant difference in a given confidence interval, and further tests %of  $H_1$ and $H_2$ can not make the decision which one group of metrics is better(worse) in the given confidence %interval, we regard the performance of two group of metrics are the same. But this situation never happen in this %experiment.


\section{Experiment Results}
In this section, we show the results we achieved in this experiment. First, we demonstrate the boxplot diagrams. And then, we explain the outcomes we obtained from statistical tests.
\subsection{Boxplot Diagrams}
%remaining 12 boxplots

%\begin{figure*}[h]
%\begin{center}
%$\begin{array}{c@{\hspace{0.1in}}c@{\hspace{0.1in}}c}
%\multicolumn{1}{l}{\mbox{\bf (a)}} &
%	\multicolumn{1}{l}{\mbox{\bf (b)}} &\multicolumn{1}{l}{\mbox{\bf (a)}} \\ [-0.53cm]
%\epsfxsize=2in
%\epsffile{cm1.ps} &
%	\epsfxsize=2in
%	\epsffile{jm1.ps} & \epsfxsize=2in \epsffile{kc1.ps} \\ [0.4cm]
%\mbox{\bf (a)} & \mbox{\bf (b)} &\mbox{c}
%\end{array}$
%\end{center}
%\caption{Boxplot of MDP datasets}
%\label{fig:boxplot}
%\end{figure*}

\begin{figure}[h]
\begin{center}
  % Requires \usepackage{graphicx}
  \includegraphics[scale=0.5]{page1.ps}%{onePage.ps}
  \caption{Boxplots of MDP datasets}\label{fig:boxp1}
\end{center}
\end{figure}

\begin{figure}[h]
\begin{center}
  % Requires \usepackage{graphicx}
  \includegraphics[scale=0.5]{page2.ps}%{onePage.ps}
  \caption{Boxplots of MDP datasets}\label{fig:boxp2}
\end{center}
\end{figure}

Figures~\ref{fig:boxp1} and~\ref{fig:boxp2} show the boxplot diagrams of 12 MDP datasets (PC5 is shown in Figure~\ref{fig:boxpc5}). From these boxplots, we have the following observations:
\bi
  \item The performances of $all$ metrics are generally better than $code$ metrics in the MDP datasets.
  \item The performances of $all$ metrics are better than $design$ metrics in all 13 MDP datasets.
  \item The performances of $code$ metrics are better than $design$ metrics in 12 datasets except in KC4 dataset.
\ei


\subsection{Statistical Test}

\begin{table}[h]
\centering
\tiny
\caption{Result of Wilcoxon test.}
\label{tab:stattest}
\begin{tabular}{|l|l|l|l|l|}
\hline & \multicolumn{3}{|c|}{chosen hypothesis and p\_value} & \\
\hline dataset & all\_code & all\_design & code\_design & trend \\
\hline \multirow{2}{*}{cm1} & H1 & H1 & H1 & \multirow{2}{*}{all>code>design}               \\
& 0.000977 & 0.000977 & 0.000977 & \\
\hline \multirow{2}{*}{kc1} & H0 & H1 & H1 & \multirow{2}{*}{all=code>design}              \\
& 0.232422 & 0.000977 & 0.000977 & \\
\hline \multirow{2}{*}{kc3} & H0 & H1 & H1 & \multirow{2}{*}{all=code>design}               \\
& 0.556641 & 0.000977 & 0.000977 & \\
\hline \multirow{2}{*}{kc4} & H1 & H1 & H2 & \multirow{2}{*}{all>design>code}               \\
& 0.000977 & 0.006836 & 0.000977 & \\
\hline \multirow{2}{*}{mc1} & H0 & H1 & H1 & \multirow{2}{*}{all=code>design}               \\
& 0.556641 & 0.000977 & 0.000977 & \\
\hline \multirow{2}{*}{mc2} & H0 & H1 & H1 & \multirow{2}{*}{all=code>design}               \\
& 0.695312	   & 0.000977 & 0.000977 & \\
\hline \multirow{2}{*}{mw1} & H1 & H1 & H1 & \multirow{2}{*}{all>code>design}               \\
& 0.013672 & 0.000977 & 0.000977 & \\
\hline \multirow{2}{*}{jm1} & H1 & H1 & H1 & \multirow{2}{*}{all>code>design}               \\
& 0.000977 & 0.000977 & 0.000977 & \\
\hline \multirow{2}{*}{pc1} & H0 & H1 & H1 & \multirow{2}{*}{all=code>design}               \\
& 0.105469 & 0.000977 & 0.000977 & \\
\hline \multirow{2}{*}{pc2} & H0 & H1 & H1 & \multirow{2}{*}{all=code>design}               \\
& 0.921875 & 0.000977 & 0.000977 & \\
\hline \multirow{2}{*}{pc3} & H0 & H1 & H1 & \multirow{2}{*}{all=code>design}               \\
& 0.083984 & 0.000977 & 0.000977 & \\
\hline \multirow{2}{*}{pc4} & H1 & H1 & H1 & \multirow{2}{*}{all>code>design}               \\
& 0.000977 & 0.000977 & 0.000977 & \\
\hline \multirow{2}{*}{pc5} & H1 & H1 & H1 & \multirow{2}{*}{all>code>design}               \\
& 0.000977 & 0.000977 & 0.000977 & \\
\hline
\end{tabular}
\end{table}

According to Demsar's recommendation~\cite{demsar06}, we first use the Friedman test to test whether there is difference in the performance among 3 groups of metrics over the 13 MDP datasets. We have the $p-value$ of $0.00003604$( $<0.05$) for the Friedman test which strongly indicates there is difference among three groups of metrics over 13 MDP datasets. Then, we use Wilcoxon test to conduct pairwise tests on the $3$ different groups in each dataset, that is, three tests are conducted in each dataset: (1) the performance of the best learner on metrics $all$ $vs.$ the performance of best learner on $code$ metrics; (2) the performance of the best learner on $all$ metrics $vs.$ that of $design$ metrics; and the performance of the best learner on $code$ metrics $vs.$ that of $design$ metrics. Table~\ref{tab:stattest} shows the test result of the Wilcoxon test and all the hypotheses which make the decision in 95\% confidence interval and their corresponding $p-value$ are shown in the table.  From the Table~\ref{tab:stattest}, we can see all the observations from the boxplot diagrams hold while painting a more detailed picture.

\bi
  \item In 7 datasets, the performances of $all$ metrics do not have any significant different from that of $code$ metrics in 95\% confidence interval.
  \item In other 6 datasets, the performances of $all$ metrics are significantly better than that of $code$ metrics in 95\% confidence interval.
  \item In all 13 datasets, the performances of $all$ metrics are better than that of $design$ metrics in 95\% confidence interval.
  \item In 12 datasets, the performances of $code$ metrics are better than that of $design$ metrics in 95\% confidence interval.
  \item In KC4 dataset, the performance of $design$ metrics are better than that of $code$ metrics in 95\% confidence interval.
\ei

In 13 MDP datasets, there are 3 groups of metrics resulting 39 pairwise tests. Table~\ref{tab:stattest} shows the test results from the Wilcoxon nonparametric test. We also conduct the Mann-Whitney test. The MannWhitney test agrees with the Wilcoxon test in all but $1$ of the 39 cases: (1) in PC2, the Mann-Whitney test concludes that $all>code$ while the Wilcoxon has $all=code$(refer to Table~\ref{tab:stattest}).


\subsection{Discussion}
To our knowledge, this is the first work that compares the performances of predictive models built based on design metrics versus code metrics. The results, although encouraging, do not come without caveats.

%First, the design metrics used here are extracted from source code by using a reverse engineering tool (McCabe IQ 7.1) which may generate different %results than gathering design metrics directly from a software project during the design stage. There are many reverse engineering tools available in %software research community~\cite{CanforaHarman07}, but currently there is no consensus recommendation for a preferred tool. Naturally the range of %available reverse engineering tools will provide different groups of metrics which presumably have an impact on the fault prediction process.

Second, in comparing the performances of different groups of metrics, only the best learner from five algorithms is selected for each group, this may not provide the best suggestion for future selection of learners if we are given a new dataset or a new group of metrics.  Referring back to Figures \ref{fig:boxp1} and \ref{fig:boxp2}, there is no algorithm that performs best across all datasets given any of the three metrics considered.  For instance, in Figure \ref{fig:boxp1}, if we look only at $all$ metrics, we see that Random Forests performs best in CM1, MC2, and KC1, but it is outperformed by NaiveBayes and Boosting in KC3 and MW1 respectively.  Taking a slightly different approach, we may wish to match an algorithm to a particular dataset with the hopes that it will perform best in all three metrics.  Referring back to Figure \ref{fig:boxp2}, we see this is the case in PC1, but fails to hold in the remaining datasets.  It is important to note however, in some cases such as PC3 and PC4 the lack of a consensus winner across all three metrics may only be due to a very small increase in performance by one learner (note the tight distributions in the boxplots).  Taking these considerations into play, one would have to carefully evaluate how to best approach a new (unsupervised) dataset both in terms of use of metrics and chosen algorithm.

Lastly, an interesting phenomenon is that the performance of predictive models built from MDP datasets are influenced more by the characteristics of different group of metrics than that of by using the different types of learning algorithms. That said, we do not currently know specifically which metrics make an overall class of metrics (code, design, etc.) more important than another class.


\section{Related Work}

One of the earliest works to study design
metrics was conducted by Ohlsson and Alberg that predicted fault-prone modules prior
to coding in Telephone Switches system of $130$ modules at Ericsson
Telecom AB~\cite{Ohlsson96}. Their design metrics are derived from graphs where functions and subroutines in a module are represented by one or more graphs. Their graphs are called Formal
Description Language ($FDL$) graphs that generate a set of direct and indirect metrics based on measures of complexity. The examples of direct metrics are the number of branches, the number of graphs in modules, the number of possible connections to a graph, and the number of paths from input signal to the output signals etc. The indirect metrics are the metrics calculated from the direct metrics using mathematical formula, such as McCabe cyclomatic complexity etc.

The suite of object oriented($OO$) metrics,referred as CK metrics,
is first proposed by Chidamber and Kemerer in~\cite{Chidamber94}.
They propose six CK design metrics including Weight Method Per Class (WMC), Number of Children (NOC),Depth of Inheritance Tree (DIT),Coupling Between Object class(CBO), Response For a Class (RFC), and Lack of Cohesion in Methods (LCOM).
Basili et al~\cite{Basili96} were among the first to validate these
CK metrics by using 8 C++ systems developed by students. They
demonstrate the usefulness of CK metrics over code metrics. In 1998, Chidamber, Darcy and Kemerer explore the relationship
between the CK metrics to productivity, rework effort or design
effort separately~\cite{Chidamber98}. They show that CK metrics have better explanatory
power traditional code metrics based on three economic variables.

%\cite{CanforaHarman07,Briand06,Antoniol01,Antoniol02,Systa00,Tonella05}
It is a hot topic in software reverse engineering to recover design from source code~\cite{CanforaHarman07,Antoniol01,Antoniol02}. Systa~\cite{Systa00} recover UML diagrams from source code using static and dynamic code analysis. Tonella and Potrich~\cite{Tonella05} extract sequence diagrams from source code through static analysis on data flow. Briand $et$ $al$. recover sequence diagrams, conditions, message exchanged and data flow from Java code by using transformation techniques~\cite{Briand06}.

There have been a few recent works, such as
\cite{Schroter06,Zimmermann07,Nagappan06,Subramanyam03}, that predict
software fault-proneness using(combining) metrics from design phase.
The metrics are either extracted from design documents or mining from source code using reverse engineering techniques.

Subramanyam and Krishnan investigate to use a subset of CK metrics
to predict software faults~\cite{Subramanyam03}. The subset of
metrics they use are WMC, CBO and DIT. The system they study is a
large B2C e-commerce application suite developed using C++ and Java.
They show that OO metrics are significantly related to defects by
using least square regression and their relationship to defect is
strongly related to language.

A recent work to apply reverse engineering to recover design metrics from source code to predict fault-proneness information is conducted by Schroter, Zimmermann, and Zeller~\cite{Schroter06}.
Schroter, Zimmermann, and Zeller predict fault-proneness information
from design stage \cite{Schroter06}. They use 52 ECLIPSE
\cite{Schroter06} plug-ins and find usage relationships between these
components as well as past failure data.   The relationships
they investigate are the usage of import statements within a single
release. The past failure data is the number of failures for a
single release.  They collect their data from version archives (like
CVS) and bug tracking systems like BUGZILLA. They build predictive models using the set of imported classes of each
file as independent variables to predict the number of failures of
the file. They also predict fault-proneness from package level where
the set of imported packages of each package is the independent
variable; the number of failures of each package is dependent
variable. In file level, the average prediction accuracy of the top 5\% is approximately 70\%; in the package level, the average prediction accuracy of the top 5\% is approximately
90\%.  Here the package level is better than that of
file level. In~\cite{Zimmermann07}, Zimmermann, Premraj and Zeller further investigate ECLIPSE open resource, they extract OO metrics along
with static code complexity metrics from ECLIPSE and point out their
effectiveness to predict software fault-proneness. Their metrics
data is posted in PROMISE~\cite{PROMISE07}. Neuhaus, Zimmermann,Holler and Zeller examine Mozilla code to extract the relationship of imports and function calls to predict software components' vulnerability ~\cite{Neuhaus07}.

Nagappan, Ball and Zeller in~\cite{Nagappan06} predict component
failures using OO metrics in five Microsoft software systems.
%The predictor variables have the following: module metrics (\# of
%classes, \# of function, \# of global varaibles), pre-function
%metrics such as the number of lines,parameters, arcs, blocks,
%readcoupling, writecoupling, AddrTakenCoupling, procCoupling, FanIn,
%FanOut, McCabe's complexity and Preclass metrics such as the number
%of classMethods, InheritanceDepth, ClassCoupling and SubClasses etc.
%The predictors they used mostly are OO metrics.
Their results show
that these metrics are suitable to predict software defects. They
also show that the predictors are only useful to predict the same or
similar projects that is also demonstrated by Menzies $et$ $al$
in~\cite{menzies04a}.


\section{Summary and Future Work}
The goal of this paper is to compare the performance of predictive models which use design-level metrics with those that use code-level metrics.  We analyze thirteen datasets from NASA MDP which offer design as well as code metrics. We find that there is a general trend in MDP datasets that the performance of $all$ metrics is better than that of $code$ metrics; and the performance of $design$ metrics is the worst of all among three. We observe an interesting phenomenon: the performance of predictive models vary more from using different group metrics than from using different learners. That is, the performances of different groups of metrics on a specific learner for a given dataset result in much larger performance gaps than that of different learners on a chosen group of metrics for a given dataset.

Using a range of modeling techniques and statistical significance tests, we confirmed that models built from code metrics typically outperform design metrics based models.  However, both types of models prove to be useful the combination of the two classes of metrics often result in statistically significant increased in fault prediction performance.  Additionally, they can be constructed in different project phases, as design metrics are available early in the software lifecycle phase which may be an important guide to improving software quality. Code-based models can be used to further increase the performance of design-level models and, thus, increase the efficiency of assigning verification and validation activities late in the development. This may have great potential to improve the software quality of safety related mission projects, i.e. NASA projects.

As touched on in the discussion section, we have yet formally establish whether the increase in performance associated with $all$ metrics is primarily the effect of adding $design$ metrics, $other$ metrics, or a mix of both.  This cause should be further investigated to determine the degree in which the inclusion of $other$ metrics is beneficial / necessary.

The notion of feature subset selection was not formally considered in this work.  Although the authors do not feel it would have significant impact on the Random Forest algorithm, the potential presence of noisy / confounding attributes may impact the performance of the other algorithms considered (in varying degrees).  Even if a complete lack of noise is assumed, surely there is a lack of complete orthogonality between attributes within a class of metrics resulting in potential redundancy.  Furthermore, it is possible that there may have relationships in attributes across the different classes of metrics.  For instance, the attributes of $NUMBER\_OF\_LINES$ which falls within the class of $code$ metrics is undoubtedly related to the $BRANCH\_COUNT$ and $CONDITION\_COUNT$ attributes in the $design$ metric class.  The degree in which each of these individual attributes contribute to prediction performance can be investigated rigorously and may yield different results if attributes are appropriately combined / removed.

%\end{document}  % This is where a 'short' article might terminate

%ACKNOWLEDGMENTS are optional
%
% The following two commands are all you need in the
% initial runs of your .tex file to
% produce the bibliography for the citations in your paper.
\bibliographystyle{abbrv}
\bibliography{pf}  % sigproc.bib is the name of the Bibliography in this case
% You must have a proper ".bib" file
%  and remember to run:
% latex bibtex latex latex
% to resolve all references
%
% ACM needs 'a single self-contained file'!
%
%APPENDICES are optional
%\balancecolumns
\end{document}
