\documentclass{sig-alternate}

%\usepackage[none]{hyphenat} 

\usepackage[table]{xcolor}
\usepackage{graphicx}
\newcommand{\G}{\cellcolor[rgb]{0.8,0.8,0.8}}

\begin{document}
%
% --- Author Metadata here ---
\conferenceinfo{ICSE}{2011 Waikiki, Honolulu, Hawaii USA}
\CopyrightYear{2011} % Allows default copyright year (20XX) to be over-ridden - IF NEED BE.
%\crdata{X-XXXXX-XX-X/XX/XX}  % Allows default copyright data (0-89791-88-6/97/05) to be over-ridden - IF NEED BE.
% --- End of Author Metadata ---

%\title{An Empirical Study on the Influence of Evaluation Criteria used in Software Cost Estimation  }
\title{Where is the best effort estimator? }

%\numberofauthors{4}
%\author{
% You can go ahead and credit any number of authors here,
% e.g. one 'row of three' or two rows (consisting of one row of three
% and a second row of one, two or three).
%
% The command \alignauthor (no curly braces needed) should
% precede each author name, affiliation/snail-mail address and
% e-mail address. Additionally, tag each line of
% affiliation/address with \affaddr, and tag the
% e-mail address with \email.
%
% 1st. author
%\alignauthor
%Author1
% 2nd. author
%\alignauthor
%Author2
% 3rd. author
%\alignauthor 
%Author3
%\and  % use '\and' if you need 'another row' of author names
% 4th. author
%\alignauthor 
%Author4

%}


\numberofauthors{2} %  in this sample file, there are a *total*
% of EIGHT authors. SIX appear on the 'first-page' (for formatting
% reasons) and the remaining two appear in the \additionalauthors section.
%
\author{
% You can go ahead and credit any number of authors here,
% e.g. one 'row of three' or two rows (consisting of one row of three
% and a second row of one, two or three).
%
% The command \alignauthor (no curly braces needed) should
% precede each author name, affiliation/snail-mail address and
% e-mail address. Additionally, tag each line of
% affiliation/address with \affaddr, and tag the
% e-mail address with \email.
%
% 1st. author
\alignauthor Jacky W. Keung\\
    \affaddr{National ICT Australia Ltd. Australia}\\
    \affaddr{CSE, University of New South Wales, Australia}\\
    \email{Jacky.Keung@nicta.com.au}
\\
\alignauthor  Ekrem Kocaguneli, Tim Menzies\\
            \affaddr{CS \& EE, WVU, Morgantown, USA}\\
       \email{ekocagun@mix.wvu.edu, tim@menzies.us }
}



\maketitle
\begin{abstract}
Background: Prior work on software cost estimation studies reported conflicting results as to which evaluation criteria such as MMRE should be used for effective evaluation of model prediction performance. It is becoming an eminent issue in the research of cost estimation models.
\\
Aim: 
\\
Method: 9 preprocessors combined with 10 learners on 20 publicly available software project datasets to produce a massive 1,800 predictions to indicate the efficacy of different evaluation criteria. 
\\
Results: Instead championing a particular evaluation criterion, our results clearly indicate the evaluation criteria being used are practically producing similar results.
\\
Conclusion: based on our finding, our recommendation is that the quality of the dataset and the preprocessor used will have an influential impact on the prediction result, different evaluation criteria examined in the study should not be used to fully appreciate the quality of the prediction model.    
\end{abstract}

% A category with the (minimum) three required fields
%\category{H.4}{Information Systems Applications}{Miscellaneous}
%A category including the fourth, optional field follows...
%\category{D.2.8}{Software Engineering}{Metrics}[complexity measures, performance measures]
\category{Software Engineering}{Software Metrics}{Data Mining}

\terms{Software Cost Estimation, Linear Regression, Regression Trees, Neural Nets, Analogy, MMRE, Evaluation Criteria}

\keywords{Software Cost Estimation, MMRE, Evaluation Criteria}

\section{Introduction}

Accurate cost estimation is extremely important in software engineering, and it is often a challenging task for many software project managers. Both overestimating and underestimating would result unfavourable impacts to the business and project resource planning. As a result, researchers have proposed and evaluated many different prediction systems derived from statistics, machine learning and data mining domains.

Building a software cost estimation model requires extensive information such as the productivity of the development environment, various different cost factors and vice versa, these information are usually present in software project datasets, and can be used to build different cost prediction models. For example, the linear regression approach from statistics and case-based reasoning (analogy) from machine learning, these techniques require company historical project data to build predictive models. Therefore the dataset quality plays an important role in the model prediction performance, and different prediction techniques can be applied in different situations depends on the amount of project data and information available. Model performance evaluation criteria becomes important to compare different model prediction performance. 

Foss et al. \cite{foss03} conducted an simulation study of the model evaluation criteria, examined the most commonly used evaluation criteria used in software cost estimation, including the de facto criterion MMRE. Their results have shown that MMRE is not recommended for comparing different model prediction performance, and call for the development of a new, simple-to-use, universally accepted metric  for evaluating prediction models. At the time of writing, the {\em ``Holy Grail''} kind of evaluation metric has not been developed in the research community. MMRE and Pred(25) still, are the most commonly used evaluation criterion for comparing the performance of different prediction models. 

The nature and the drawbacks of these evaluation criteria are well studied, and used for the purpose of comparing different software prediction techniques. For example Shepperd and Kadoda \cite{shepperd01b} considered that none of these existing prediction models proven consistently accurate, and the accuracy of an estimate depends on the dataset characteristic and a suitable prediction model for the dataset. Their conclusion was that is generally infeasible to determine which prediction technique is "best".   This raises the issues as to how to select the best effort estimator under different situations or datasets? 

This study attempts to survey the prediction results produced from a number of datasets using a large number different prediction methods. That is 9 data preprocessors combined with 10 different learners on 20 publicly available software project datasets to produce a massive 1,800 software estimates. Results are completely unexpected, not only we have found that CART (Classification and Regression Tree) method consistently outperforms other methods, we also found that different evaluation criteria are less important than we anticipated. In addition, the quality and characteristics of the dataset is very important, as data quality controls the prediction accuracy, therefore a data preprocessor also plays an significant role in the estimation process. 

The consequences of our findings cast doubts on the results of all studies claiming one model is better than the others without investigating the quality and characteristics of the datasets being used. Our result is novel and important for the research in software cost estimation, provides evidence showing that a "best" effort estimator can be identified, and showing recommendations on dataset certification before it is being applied for cost estimation. 

The paper is organized as follows: Section 2 provides an summary of related work with a discussion of MMRE. Section 3 provides our theoretical hypothesis. Section 4 presents our experiment design follow by the execution of the experiments in Section 5. Results are presented in Section 6 followed up by a discussion in Section 7. Section 8 concludes and summaries the paper. 





\section{Related Work}

Evaluation of the evaluation metrics themselves seems to have received little attention since Conte et al. [13] publicized MMRE and other measures. The only related work we are aware of is Miyazaki et al. [38], Kitchenham et al. [31], Foss et al. [20], and Stensrud et al. [50].

The Mean Magnitude of Relative Error measure a.k.a. MMRE is the most commonly used evaluation criterion for selecting the best effort estimator from a number of competing software prediction models\cite{shepperd97}\cite{foss03}. MMRE measures the error ratio between the actual effort and the predicted effort and can be expressed as the following equation:  
{\begin{equation}\label{one}
MRE={\mid{x-\hat{x}}\mid\over {x}}
\end{equation}}

Foss et al.\cite{foss03} have provided an extensive discussion using a simulation study demonstrating that MMRE is not reliable and does not select the best model, and their recommendation was using a combination of theoretical justification of the models. 

Alternative evaluation metrics are Pred(25)  



MER (Magnitude of error relative to the estimate)

RSD (Relative standard deviation)

LSD (Logarithmic standard deviation)

BRE (Balanced relative error)

IBRE(Inverted balanced relative error)




Wilcoxon-Mann-Whitney test is a non-parametric statistical hypothesis test commonly used to compare the difference between two groups of sample data, and it is used as an alternative to the paired Student's t-test when the population distribution cannot be assumed to be normal. It is more robust than the Student's t-test as it compares the sums of ranks, unlike Student's t-test which may introduce spurious findings as a result of presence of outliers may be existed in the given datasets. 



\subsection {Comparing effort estimators}

There have been attempts in trying to compare the model prediction performance of different effort estimation models. Shepperd and Kododa \cite{shepperd01b} compared regression, rule induction, nearest neighbour and neural nets, and it was suggesting the characteristics of the datasets

Shepperd and Kadoda attempted to explore the relationship between accuracy, choice of prediction system, and different dataset characteristic by using a simulation study. 


\subsection {Mean Magnitude of Relative Error}

Mean Magnitude of Relative Error (MMRE) is probably the most widely used evaluation criterion for assessing the performance of competing software prediction models. However it has flaws.


\section{Hypothesis}

\section{Experiment Design}

Ekrem

list 10-preprocessors
      9 learners
   


\input {ekrem/pre-processor}
\input {ekrem/learner}

\section{Datasets}

Table of the ranked 20 datasets here >
Telecome
Kermerer
cocomo81o

Homogenization 

\section{Result}



\subsection{Dot plots}

The resulting Win/Loss/Ties values have been computed for all different combination of preprocessor+learner. To summarise the result, the number of losses of each combination are shown in Figure \ref{fig:dot-plot}. 

\begin{figure*}
\begin{center}
\includegraphics[width=3.5in]{lib/thinks.pdf} \end{center}
\caption[Dot-Plot]{Overall estimation results using different methods, ranked by the number of losses.} \label{fig:dot-plot}
\end{figure*}

20(datasets) x 9 (pre-processors) x 10(learners) = 1,800 results were generated, they were then ranked according to the number of losses using the Wilcoxon-Mann-Whitney test, and classified into 4 different groups of percentage of losses, indicated by 4 different colours. The {\em x-axis} shows 20 datasets used in the study, the {\em y-axis} presents different method combinations (pre-processor+learner), due to limited space, they are not labeled on the graph. Table \ref {table:res_rank} provides a detailed summary of all different method combinations, which also represents the corresponding labels of the {\em y-axis} shown in Figure \ref{fig:dot-plot}.

In all 90 method combinations, CART (Classification and Regression Tree) had the least losses, that is, CART consistently achieves statistically significant lower MRE values of all the methods using different datasets in the study. For top 10 method combinations, CART as a learner outperformed all other prediction models, such as Analogy and Linear Regression Analysis. 

In addition to the given results shown in Table \ref{table:freq_method}, there are some interesting findings deserve some attention. Firstly, as mentioned above, CART performed consistently better than any other methods as a learner, this is evident in the learner section of Table \ref{table:freq_method}, where CART (yes) and CART (no) ranked in the top according to the least losses in the race. Interestingly, the worst learners are the Regression methods (SWR, and LR) and the NeuralNet method (NNet).

Although SWR (stepwise regression) as a learner did not perform well, but as a pre-processor in the prediction system it is significantly important, of the 90 different combinations, SWR leads to least losses combining with different learners. This is evident by its occurrence indicated in Table \ref{table:freq_method}. The 2nd most important data pre-processor is SFS (Stepwise Forward Selection). The implication in here is that combining SWF or SFS with any other learners will result in favourable prediction accuracy, as both stepwise procedures (SWR and SFS) are likely to statistically filter only the relevant project features for the learners to predict estimates. The finding coincides with Keung et al.'s work \cite{keung08a}, which argues the importance of a data preprocessor in software cost estimation, in their experiments, a stepwise feature selection system similar to that of SWR was adopted, results showing improved prediction performance as well. 

Finally, a very important finding can be identified relating to the evaluation criteria MMRE. Our results show that regardless of the evaluation criteria used the ranking of different methods virtually shows no difference, the implication is that different evaluation criteria can be used, they all arrives the same conclusion that is CART as a learner together with a stepwise data pre-processor (SWR or SFS) will consistently perform the best. Our results also shows that the differences using different evaluation criteria are statistically insignificant. This leads to the end of a continouse debate as to whether we should continue to use MMRE or an alternative metric for comparing different prediction models. 

From the dot-plot graph we can see that the "world is flat", especially based on the first few datasets such as telecom, kemerer, cocomo81o. The implication is that using different prediction models may result similar findings in most cases, again this is largely depends upon on the dataset characteristics and quality. In this study, dataset China is a cross-company large dataset, performed worst among the 20 datasets, nevertheless, CART learner is still the preferred method which result in least losses. 


 




\begin{table*}
\small

{\scriptsize
{\renewcommand{\baselinestretch}{.5}
\begin{tabular}{rrr|rrr|rrr}
rank&prep&learner&rank&prep&learner&rank&prep&learner\\\hline
1&norm&CART (yes)&31&freq5bin&CART (no)&61&freq3bin&CART (no)\\
2&norm&CART (no)&32&width5bin&CART (yes)&62&PCA&K=1\\
3&none&CART (yes)&33&width5bin&CART (no)&63&width3bin&SWR\\
4&none&CART (no)&34&norm&K=5&64&width5bin&PLSR\\
5&log&CART (yes)&35&PCA&SWR&65&log&SWR\\
6&log&CART (no)&36&none&K=5&66&log&PCR\\
7&SWR&CART (yes)&37&SWR&SWR&67&log&PLSR\\
8&SWR&CART (no)&38&SFS&SWR&68&width3bin&PLSR\\
9&SFS&CART (yes)&39&log&K=5&69&width3bin&K=1\\
10&SFS&CART (no)&40&norm&SWR&70&width5bin&PCR\\
11&SWR&K=1&41&none&SWR&71&norm&PCR\\
12&log&K=1&42&freq3bin&K=5&72&width3bin&PCR\\
13&SWR&K=5&43&PCA&K=5&73&freq5bin&PCR\\
14&SFS&K=5&44&width3bin&CART (yes)&74&freq5bin&SWR\\
15&PCA&PLSR&45&width3bin&CART (no)&75&width3bin&LReg\\
16&SWR&PCR&46&PCA&NNet&76&freq3bin&PCR\\
17&none&PLSR&47&width3bin&K=5&77&width5bin&LReg\\
18&SFS&K=1&48&none&NNet&78&freq3bin&PLSR\\
19&PCA&PCR&49&width5bin&SWR&79&freq5bin&PLSR\\
20&none&PCR&50&width5bin&K=1&80&log&LReg\\
21&PCA&CART (yes)&51&none&LReg&81&freq3bin&SWR\\
22&PCA&CART (no)&52&width5bin&K=5&82&freq5bin&LReg\\
23&freq5bin&K=5&53&SFS&NNet&83&width5bin&NNet\\
24&SWR&PLSR&54&norm&PLSR&84&norm&NNet\\
25&SFS&LReg&55&freq5bin&K=1&85&width3bin&NNet\\
26&norm&K=1&56&SWR&NNet&86&log&NNet\\
27&none&K=1&57&SWR&LReg&87&freq3bin&NNet\\
28&SFS&PCR&58&norm&LReg&88&freq5bin&NNet\\
29&SFS&PLSR&59&freq3bin&K=1&89&freq3bin&LReg\\
30&freq5bin&CART (yes)&60&freq3bin&CART (yes)&90&PCA&LReg\\
\end{tabular}
}
}

\caption[Result Rank]{  Detailed method combinations and their ranks for least losses 
}\label{table:res_rank}
\end{table*}


\begin{table*}
\center{

{\small
\begin{tabular}{rr|rccccc|l}\cline{3-8}
&    &      & \multicolumn{5}{c|}{Frequency of methods in ranks}\\
&    &      & 1..8& 9..30& 31..50& 51..70&71..90&    \\\cline{3-8}
&    &       CART (yes)& 4& 2& 3& 1& &    \\
&    &      CART (no)& 4& 2& 3& 1& &    \\

&&      {\G}      K=5 & {\G}  & {\G} 3 & {\G} 6 & {\G} 1 & {\G} & \\
learners                    &&   {\G}       K=1 & {\G}  & {\G} 5 & {\G}  & {\G} 5     &{\G}\\
       &              &      {\G}   PCR &  {\G} & {\G} 4 & {\G}  & {\G} 1 & {\G} 5&\\
        &              &     {\G}   PLR & {\G} 4 & {\G}  & {\G} 4 & {\G}  & {\G} 2&\\

                     &  &     LR& & 1& & 3& 6&  \\
       &  &    SWR& & & 6& 2 &2&\\
                      &   &   NNet& & & 2& 2& 6&\\\cline{3-8}

      &          & SWR&2&4&1&2&\\
&             &   SFS& &7&1&1& \\

&& {\G}  none& {\G} 2& {\G} 3& {\G} 3& {\G} 1& {\G} &\\
pre-processors          && {\G} log& {\G} 2& {\G} 1& {\G} 1& {\G} 3& {\G} 2&\\
        && {\G}   norm& {\G} 2& {\G} 1& {\G} 2& {\G} 2& {\G} 2&\\
	   && {\G} PCA& {\G} & {\G} 4& {\G} 3& {\G} 1& {\G} 1& \\

     && freq5bin&&1&2&1&5&\\
&& width2bin&&&3&3&3&\\
    &&  width5bin&&&3&3&3&\\
    && freq3bin&&&1&3&5&\\
\cline{3-8}
\end{tabular}
}

\caption[Frequency of Methods in Ranks]{  Summary of position of methods 
}\label{table:freq_method}

}
\end{table*}


\subsection{Spectrum of Pred(25) across the data sets} 
\begin{center} \includegraphics[width=3in]{lib/pred25sByData.pdf} \end{center}
\section{Spectrum of Pred(25) down the methods} 
\begin{center} \includegraphics[width=3in]{lib/pred25s.pdf} \end{center}
\section{Errors down the methods}
\begin{center} \includegraphics[width=3in]{lib/rows.pdf} \end{center}
\section{Errors across the data sets}
\begin{center} \includegraphics[width=3in]{lib/cols.pdf} \end{center}

\subsection{Comparing Pred(25) between ``best'' (at rank=1) and ``simplest'' (at rank=11)}
\begin{center}
{\scriptsize
\begin{tabular}{r|rr|r}
  &\multicolumn{3}{c}{PRED(25)}\\\cline{2-4}
&norm&log& \\
data&CART(yes)&K=1& difference\\\hline
china&95&43&-52\\
sdr&42&17&-25\\
desharnaisL2&48&32&-16\\
nasa93\_center\_1&58&42&-16\\
cocomo81e&18&4&-14\\
desharnais&43&37&-6\\
nasa93\_center\_5&36&33&-3\\
maxwell&32&31&-1\\
albrecht&33&33&0\\
miyazaki94&40&40&0\\
desharnaisL1&39&41&2\\
cocomo81o&29&33&4\\
finnish&61&66&5\\
cocomo81&13&22&9\\
cocomo81s&9&18&9\\
nasa93&29&41&12\\
kemerer&7&20&13\\
nasa93\_center\_2&43&59&16\\
desharnaisL3&20&40&20
\end{tabular}}\end{center}




\section{Discussion}



\subsection {Data Certification}



\section{Conclusion}

\section{Acknowledgement}

\bibliographystyle{abbrv}
\bibliography{icse} 
% You must have a proper ".bib" file
%  and remember to run:
% latex bibtex latex latex
% to resolve all references
%
% ACM needs 'a single self-contained file'!
%

\end{document}
