\documentclass{sig-alternate}
\usepackage{times}
\usepackage{algorithm}
\usepackage{algorithmic}



%\usepackage[none]{hyphenat} 
\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{itemize}
  \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{itemize}}

\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}


\usepackage[table]{xcolor}
\usepackage{url,graphicx}
\newcommand{\G}{\cellcolor[rgb]{0.8,0.8,0.8}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\hyp}[1]{Hypothesis~\ref{hyp:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}

\newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
\newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}

\begin{document}
%
% --- Author Metadata here ---
\conferenceinfo{ICSE}{2011 Waikiki, Honolulu, Hawaii USA}
\CopyrightYear{2011} % Allows default copyright year (20XX) to be over-ridden - IF NEED BE.
%\crdata{X-XXXXX-XX-X/XX/XX}  % Allows default copyright data (0-89791-88-6/97/05) to be over-ridden - IF NEED BE.
% --- End of Author Metadata ---

%\title{An Empirical Study on the Influence of Evaluation Criteria used in Software Cost Estimation  }
\title{Where is the best effort estimator? }

%\numberofauthors{4}
%\author{
% You can go ahead and credit any number of authors here,
% e.g. one 'row of three' or two rows (consisting of one row of three
% and a second row of one, two or three).
%
% The command \alignauthor (no curly braces needed) should
% precede each author name, affiliation/snail-mail address and
% e-mail address. Additionally, tag each line of
% affiliation/address with \affaddr, and tag the
% e-mail address with \email.
%
% 1st. author
%\alignauthor
%Author1
% 2nd. author
%\alignauthor
%Author2
% 3rd. author
%\alignauthor 
%Author3
%\and  % use '\and' if you need 'another row' of author names
% 4th. author
%\alignauthor 
%Author4

%}


\numberofauthors{2} %  in this sample file, there are a *total*
% of EIGHT authors. SIX appear on the 'first-page' (for formatting
% reasons) and the remaining two appear in the \additionalauthors section.
%
\author{
% You can go ahead and credit any number of authors here,
% e.g. one 'row of three' or two rows (consisting of one row of three
% and a second row of one, two or three).
%
% The command \alignauthor (no curly braces needed) should
% precede each author name, affiliation/snail-mail address and
% e-mail address. Additionally, tag each line of
% affiliation/address with \affaddr, and tag the
% e-mail address with \email.
%
% 1st. author
\alignauthor Jacky W. Keung\\
    \affaddr{National ICT Australia Ltd. Australia}\\
    \affaddr{CSE, University of New South Wales, Australia}\\
    \email{Jacky.Keung@nicta.com.au}
\\
\alignauthor  Ekrem Kocaguneli, Tim Menzies\\
            \affaddr{CS \& EE, WVU, Morgantown, USA}\\
       \email{ekocagun@mix.wvu.edu, tim@menzies.us }
}



\maketitle
\begin{abstract}
Background: Prior work on software effort estimation studies reported conflicting results as to which evaluation criteria such as MMRE should be used for effective evaluation of model prediction performance. It is becoming an eminent issue in the research of effort estimation models.
\\
Aim: 
\\
Method: 9 preprocessors combined with 10 learners on 20 publicly available software project datasets to produce a massive 1,800 predictions to indicate the efficacy of different evaluation criteria. 
\\
Results: Instead championing a particular evaluation criterion, our results clearly indicate the evaluation criteria being used are practically producing similar results.
\\
Conclusion: based on our finding, our recommendation is that the quality of the dataset and the preprocessor used will have an influential impact on the prediction result, different evaluation criteria examined in the study should not be used to fully appreciate the quality of the prediction model.    
\end{abstract}

% A category with the (minimum) three required fields
%\category{H.4}{Information Systems Applications}{Miscellaneous}
%A category including the fourth, optional field follows...
%\category{D.2.8}{Software Engineering}{Metrics}[complexity measures, performance measures]
\category{Software Engineering}{Software Metrics}{Data Mining}

\terms{Software effort Estimation, Linear Regression, Regression Trees, Neural Nets, Analogy, MMRE, Evaluation Criteria}

\keywords{Software Effort Estimation, MMRE, Evaluation Criteria}

\section{Introduction}

Accurate effort estimation is vital in software
engineering, and is often a challenging task for many software
project managers. Both overestimating and underestimating would
result unfavourable impacts to the business and project resource
planning. Hence,  researchers have proposed and evaluated many
different prediction systems derived from statistics, machine
learning and data mining domains.

Building a software effort estimation model requires extensive
information such as development environment productivity and
numerous other effort factors.  This information
can be used
to build different effort prediction models. For example, the linear
regression approach from statistics and case-based reasoning (analogy)
from machine learning, these techniques require company historical
project data to build predictive models. 

The literature reports a large number of effort estimation methods.
For example, if we just look at instance-based methods,
\fig{cbr} lists thousands of  
algorithms  for instance-based effort estimator. 
A similiar list for model-based 
effort estimation would be just as long
(and would 
linear regression, regression trees, model-trees, neural nets, etc.).

\begin{figure}
\begin{center}
{\small
\begin{tabular}{p{0.95\linewidth}}\hline
In instance-based learning, conclusions are drawn from instances near
the test instance.
The distance measure can be constructed many ways.
Mendes et al.~\cite{Mendes2003} discuss three:
\bi
\item[$M_1$]: A simple Euclidean measure;
\item[$M_2$]: A ``maximum distance'' measure that that focuses on the single feature that maximizes
inter-project distance.
\item[$M_3$]: More elaborate kernel estimation methods.
\ei
Once the nearest neighbors are found, they must be used to generate an effort
estimate via...
\bi
\item[$R_1$]: Reporting the  median effort value of the analogies;
\item[$R_2$]: Reporting the mean dependent value;
\item[$R_3$]: Reporting a weighted mean where the nearer analogies are weighted higher than those further away~\cite{Mendes2003};
\item[$R_4 .. R_6$]: Summarize the neighborhood  with  regression~\cite{baker07},
model trees~\cite{Menzies2006} or a neural network~\cite{Li2009a}.
\ei
Prior to running  an instance-based learning, it is sometimes recommended
to handle anomalous rows by:
\bi
\item[$N_1$]: Doing nothing at all;
\item[$N_2$]: Using outlier removal~\cite{keung2008b};
\item[$N_3$]: {\em Prototype generation}; i.e. 
replace the data set with a smaller set of most representative examples~\cite{chang74}.
\ei
When computing distances between pairs,  some feature weighting scheme is often applied:
\bi
\item[$W_1$]: All features have uniform weights;
\item[$W_2.. W_9$]: Some pre-processing scores the relative value of the features. 
Keung~\cite{keung2008b}, Li et al.~\cite{Li2009}, and Hall \& Holmes~\cite{hall03} review eight different pre-processors.
Note that these pre-processors may require  {\em discretization} (discussed below).
\ei
Discretization breaks up  continuous ranges
at  points $b_1,b_2,...$, each containing  counts of  $c_1,c_2,...$ of numbers~\cite{gama06}.
Discretization methods include:
\bi
\item[$D_1$]: Equal-frequency, where $c_i=c_j$;
\item[$D_2$]: Equal-width, where $b_{i+1} - b_i$ is a constant;
\item[$D_3$]: Entropy~\cite{FayIra93Multi};
\item[$D_4$]: PKID~\cite{YanWeb02Comparative};
\item[$D_5$]: Do nothing at all.
\ei
Finally, there is the issue of how many $k$ neighbors should be used:
\bi
\item[$K_1$]: $k=1$ is used by Lipowezky et al.~\cite{Lipowezky1998} and
Walkerden \&
Jeffery~\cite{Walkerden1999};
\item[$K_2$]: $k=2$ is used by Kirsopp \& Shepperd~\cite{Kirsopp2002}
\item[$K_3$]: $k=1,2,3$ is used by Mendes el al.~\cite{Mendes2003}
\item[$K_4$]: Li et al. use $k=5$~\cite{Li2009};
\item[$K_5$]: Baker  tuned $k$ to a particular training set using an experimental method~\cite{baker07}.
\ei\\\hline
\end{tabular}}
\end{center}
\caption{Each combination of the above
$N*W*D*M*R*K$ techniques is one {\em algorithm} for effort estimation.
This figure shows
$3*9*5*3*6*5>12,000$ algorithms for effort estimation.
Some of these ways can be ruled out, straight away.
For example, at $k=1$, then all the adaptation mechanisms
return the same result. Also, not all the feature weighting
techniques require discretization, decreasing the space of
options by a factor of five. However, even after discarding
some combinations, there are still thousands of algorithms
to explore.
 }\label{fig:cbr}
\end{figure}

One response to this very large space of estimation algorithms
is to utilize cloud computing to 
explore a large number of alternatives.
Given the ready availability of CPU farms, it is possible to create a very
large experiment tried (say) all the options of \fig{cbr}.
The main argument of this
paper is that such an analysis is {\em fundamentally misdirected} since
the properties of the effort estimation {\em data} is a much more
 important that the estimation {\em algorithms}. 

In order to support this claim,  this paper demonstrates a study of nearly one
hundred different algorithms executed on numerous data sets. 
The two stand-out results from that study are:
\bi
\item For nearly half the data sets, most algorithms perform nearly the same.
\item There exists an ordering to the remaining data sets, under which some differences
in the algorithms are detectable.
\ei
Based on this result, we conjectured that
features of the data were more important than features of the algorithms.
To formally test that conjecture, we designed a new effort estimator (called TEAK)
that focused on the the structure of the data.
The results of the TEAK study are most encouraging, the data exploration approach
of TEAK significantly out-performed standard effort estimation methods with characteristic consideration of the dataset.  

Our conclusions are two two-fold:
\be
\item We must doubt results of any
study claiming one algorithm is better than others without
investigating the {\em quality and characteristics of the datasets} being
used. 
\item
 In order to make future advances in effort estimation,
it is recommended to explore the  {\em data} quality rather than the space of possible
{\em algorithms}.
\ee
The rest of this paper is structured as follows.
The paper is organized as follows: Section 2 provides an summary of related work with a discussion of MMRE. Section 3 provides our theoretical hypothesis. Section 4 presents our experiment design follow by the execution of the experiments in Section 5. Results are presented in Section 6 followed up by a discussion in Section 7. Section 8 concludes and summaries the paper. 


Before beginning, we digress to note that the conclusions of this paper cannot be
extrapolated to 
{\em all} effort estimation algorithms and {\em all} effort estimation data sets. Given
the space of all algorithms (e.g. \fig{cbr}), such a study over all methods may be
practically impossible. 
A clear goal for future work would be to repeat this analysis on different data sets and algorithms.

Having said that, we add that this paper  focuses on algorithms that have been 
extensively studied in the 
literature~\cite{shepperd97} and the commonly available data sets (that is, the ones
available in the PROMISE repository of reuable SE data\footnote{\url{http://promisedata.org/data}}.
That is, we assert that these results should apply to much to current published literature
on effort estimation.

\input{ekrem/data}

\section{Related Work}

\input{jacky/related}
%\input{ekrem/accuracy}

\section{Hypothesis}

Software effort estimation researchers have focused on the development of advanced algorithms and optimizing models in order to achieve better prediction accuracy measured by different evaluation criteria, so that the result is comparable to many existing prediction accuracy results produced. In many cases, the prediction performance improvements are less than significant in a statistical sense. Keung \cite{keung08b} argues that estimates are probabilistic in nature, they represents the most likely values based on historical data, an error-free prediction in software cost estimation is both empirically and theoretically impossible. Using different prediction systems may result in similar outcomes only with a small variation in the prediction accuracy, this is because different quality and characteristics of the dataset determine the {\em Theoretical Maximum Prediction Accuracy} (TMPA)\cite{keung08b} of any prediction system being used. Some prediction system may be more suitable than the others on a dataset, this is because the  algorithm within a prediction system is more suitable for the dataset characteristic. In Keung \cite{keung08b}'s experiment, to optimize the prediction accuracy, one approach is to dynamically select a method that would produce a favourable result given the actual effort is known for comparison, he used a different number of k-nearest neighbours for each data point estimate, resulting in an improved overall performance accuracy using the entire dataset. It shows that variance in the dataset drastically changed the prediction accuracy, rather than using different prediction models.  

The dataset quality and their characteristics are usually overlooked in the development of a better algorithm for software effort estimation. Studies shown the use of a data preprocessor in the estimation experiment generally show a significant improvement in the estimation accuracy. Dataset homogenization will also result in improved performance in estimation. \cite{mendes_04}\cite{kitchenham_07}

Without looking into the eminent issue of data quality and their characteristics, the research into the development of a better effort estimator had reached its destination. 
If the above statement sustains, then the selection of a single useful evaluation criteria and the prediction algorithm are less important than the dataset quality itself. Research effort should be more focusing on the evaluation of dataset quality and its characteristics, including variance in the dataset. 






\section{Experiment Design}
    
This section describes the  data and algorithms used in this paper.
As mentioned in the introduction, it is impractical to study {\em all} effort estimation methods
since there are too many of them. The following data sets and algorithms were selected since
we can see their widespread use in the effort estimation literature.

\subsection{Datasets}

Figure \ref {fig:data} summarizes the datasets that were used for our comparison study using different data processors and learner algorithms. As can be seen from the list the datasets, they are quite diverse and were collected from many different software application domains and development environment across different continents in the world. A total of 11 publicly available datasets were used, and an additional 9 homogenised data subsets were produced from COCOMO81, NASA93 and Desharnais datasets to examine the effects of homogeneity and its influence on the prediction results.  Altogether, 20 real industry real software project datasets were used in the study. It is important to stress that none of the dataset was selected in favour of a particular prediction approach. All the datasets are available from the PROMISE repository. 


\subsection{Evaluation Method}

The study applied a Wilcoxon-Mann-Whitney signed rank test (95\% confidence) to summarize the results of the MRE and PRED(25) measures. Wilconxon is a non-parametric statistical hypothesis test commonly used to compare the difference between two groups of sample data, and it is used as an alternative to the paired Student's {\em t}-test when the population distribution cannot be assumed to be normal. It is more robust than the Student's {\em t}-test as it compares the sums of ranks, unlike Student's {\em t}-test which may introduce spurious findings as a result of presence of outliers may be existed in the given datasets. Ranked\footnote{In a ranking analysis, the raw results 
(10.2,21.3,22.1,24,25,30,100) are
replaced with their ranks in the sort order; i.e. 
(1,2,3,4,5,6,7).}
statistical tests like the Wilcoxon are also useful if it is not clear that the underlying
distributions are Gaussian~\cite{kleijnen97}.

The pseudocode for win-tie-loss calculation is given in Figure \ref {FigureWinTieLossPseudocode}.
To calculate the win-tie-loss values, we first to check if two distributions $i, j$ are statistically different according to the Wilcoxon test. Otherwise we increment $tie_i$ and $tie_j$. On the other hand, if they returned to be different, we update $win_i,win_j$ and $loss_i,loss_j$ after a numerical comparison of their median values. The following experiment will draw results based on the Wilconxon test. 

\begin{figure}
\renewcommand{\baselinestretch}{1}
\begin{algorithmic} \footnotesize
\STATE \textit{$win_{i}$} = 0, \textit{$tie_{i}$} = 0, \textit{$loss_{i}$} = 0
\STATE \textit{$win_{j}$} = 0, \textit{$tie_{j}$} = 0, \textit{$loss_{j}$} = 0
\IF{WILCOXON(\textit{$P_i$}, \textit{$P_j$}) says they are the same}
	\STATE \textit{$tie_{i}$} = \textit{$tie_{i}$} + 1;
	\STATE \textit{$tie_{j}$} = \textit{$tie_{j}$} + 1;
\ELSE
	\IF{median(\textit{$P_{i}$}) $<$ median(\textit{$P_{j}$})}
		\STATE \textit{$win_{i}$} = \textit{$win_{i}$} + 1
		\STATE \textit{$loss_{j}$} = \textit{$loss_{j}$} + 1
	\ELSE
		\STATE \textit{$win_{j}$} = \textit{$win_{j}$} + 1
		\STATE \textit{$loss_{i}$} = \textit{$loss_{i}$} + 1
	\ENDIF
\ENDIF
\end{algorithmic}
\caption{Pseudocode for Win-Tie-Loss Calculation Between Variant \textit{i} and \textit{j}
with performance measures $P_i$ and $P_j$.}
\label{FigureWinTieLossPseudocode}
\end{figure}





\input {ekrem/pre-processor}
\input {ekrem/learner}

\section{Result}

The resulting Win/Loss/Ties values have been computed for all different combination of preprocessor+learner. To summarise the result, the number of losses of each combination are shown in Figure \ref{fig:dot-plot}. 

\begin{figure}[!t]
\begin{center}
\includegraphics[width=3.5in]{lib/thinks.pdf} \end{center}
\caption[Dot-Plot]{Overall estimation results using different methods, ranked by the number of losses.} \label{fig:dot-plot}
\end{figure}

20(datasets) x 9 (pre-processors) x 10(learners) = 1,800 results were generated, they were then ranked according to the number of losses using the Wilcoxon-Mann-Whitney test, and classified into 4 different groups of percentage of losses, indicated by 4 different colours. The {\em x-axis} shows 20 datasets used in the study, the {\em y-axis} presents different method combinations (pre-processor+learner), due to limited space, they are not labeled on the graph. Table \ref {table:res_rank} provides a detailed summary of all different method combinations, which also represents the corresponding labels of the {\em y-axis} shown in Figure \ref{fig:dot-plot}.



\subsection{The World is ``Flat''}

We have observed over 50\% of the results have Wilcoxon losses less than 12.5\%, only a small fraction of the results with losses greater than 50\%.
In all 90 method combinations, CART (Classification and Regression Tree) had the least losses, that is, CART consistently achieves statistically significant lower MRE values of all the methods using different datasets in the study. For top 10 method combinations, CART as a learner outperformed all other prediction models, such as Analogy and Linear Regression Analysis. 

In addition to the given results shown in Table \ref{table:freq_method}, there are some interesting findings deserve some attention. Firstly, as mentioned above, CART performed consistently better than any other methods as a learner, this is evident in the learner section of Table \ref{table:freq_method}, where CART (yes) and CART (no) ranked in the top according to the least losses in the race. Interestingly, the worst learners are the Regression methods (SWR, and LR) and the NeuralNet method (NNet).

Although SWR (stepwise regression) as a learner did not perform well, but as a pre-processor in the prediction system it is significantly important, of the 90 different combinations, SWR leads to least losses combining with different learners. This is evident by its occurrence indicated in Table \ref{table:freq_method}. The 2nd most important data pre-processor is SFS (Stepwise Forward Selection). The implication in here is that combining SWF or SFS with any other learners will result in favourable prediction accuracy, as both stepwise procedures (SWR and SFS) are likely to statistically filter only the relevant project features for the learners to predict estimates. The finding coincides with Keung et al.'s work \cite{keung08a}, which argues the importance of a data preprocessor in software effort estimation, in their experiments, a stepwise feature selection system similar to that of SWR was adopted, results showing improved prediction performance as well. 

Finally, a very important finding can be identified relating to the evaluation criteria MMRE. Our results show that regardless of the evaluation criteria used the ranking of different methods virtually shows no difference, the implication is that different evaluation criteria can be used, they all arrives the same conclusion that is CART as a learner together with a stepwise data pre-processor (SWR or SFS) will consistently perform the best. Our results also shows that the differences using different evaluation criteria are statistically insignificant. This leads to the end of a continouse debate as to whether we should continue to use MMRE or an alternative metric for comparing different prediction models. 

From the dot-plot graph we can see that the "world is flat", especially based on the first few datasets such as telecom, kemerer, cocomo81o. The implication is that using different prediction models may result similar findings in most cases, again this is largely depends upon on the dataset characteristics and quality. In this study, dataset China is a cross-company dataset, performed worst among the 20 datasets, nevertheless, CART learner is still the preferred method which result in least losses. 



\begin{table*}
\small
\center
{\scriptsize
{\renewcommand{\baselinestretch}{.5}
\begin{tabular}{rrr|rrr|rrr}
rank&prep&learner&rank&prep&learner&rank&prep&learner\\\hline
1&norm&CART (yes)&31&freq5bin&CART (no)&61&freq3bin&CART (no)\\
2&norm&CART (no)&32&width5bin&CART (yes)&62&PCA&K=1\\
3&none&CART (yes)&33&width5bin&CART (no)&63&width3bin&SWR\\
4&none&CART (no)&34&norm&K=5&64&width5bin&PLSR\\
5&log&CART (yes)&35&PCA&SWR&65&log&SWR\\
6&log&CART (no)&36&none&K=5&66&log&PCR\\
7&SWR&CART (yes)&37&SWR&SWR&67&log&PLSR\\
8&SWR&CART (no)&38&SFS&SWR&68&width3bin&PLSR\\
9&SFS&CART (yes)&39&log&K=5&69&width3bin&K=1\\
10&SFS&CART (no)&40&norm&SWR&70&width5bin&PCR\\
11&SWR&K=1&41&none&SWR&71&norm&PCR\\
12&log&K=1&42&freq3bin&K=5&72&width3bin&PCR\\
13&SWR&K=5&43&PCA&K=5&73&freq5bin&PCR\\
14&SFS&K=5&44&width3bin&CART (yes)&74&freq5bin&SWR\\
15&PCA&PLSR&45&width3bin&CART (no)&75&width3bin&LReg\\
16&SWR&PCR&46&PCA&NNet&76&freq3bin&PCR\\
17&none&PLSR&47&width3bin&K=5&77&width5bin&LReg\\
18&SFS&K=1&48&none&NNet&78&freq3bin&PLSR\\
19&PCA&PCR&49&width5bin&SWR&79&freq5bin&PLSR\\
20&none&PCR&50&width5bin&K=1&80&log&LReg\\
21&PCA&CART (yes)&51&none&LReg&81&freq3bin&SWR\\
22&PCA&CART (no)&52&width5bin&K=5&82&freq5bin&LReg\\
23&freq5bin&K=5&53&SFS&NNet&83&width5bin&NNet\\
24&SWR&PLSR&54&norm&PLSR&84&norm&NNet\\
25&SFS&LReg&55&freq5bin&K=1&85&width3bin&NNet\\
26&norm&K=1&56&SWR&NNet&86&log&NNet\\
27&none&K=1&57&SWR&LReg&87&freq3bin&NNet\\
28&SFS&PCR&58&norm&LReg&88&freq5bin&NNet\\
29&SFS&PLSR&59&freq3bin&K=1&89&freq3bin&LReg\\
30&freq5bin&CART (yes)&60&freq3bin&CART (yes)&90&PCA&LReg\\
\end{tabular}
}
}

\caption[Result Rank]{ Detailed method combinations and their ranks for least losses 
}\label{table:res_rank}
\end{table*}


\begin{table*}
\center{

{\small
\begin{tabular}{rr|rccccc|l}\cline{3-8}
&    &      & \multicolumn{5}{c|}{Frequency of methods in ranks}\\
&    &      & 1..8& 9..30& 31..50& 51..70&71..90&    \\\cline{3-8}
&    &       CART (yes)& 4& 2& 3& 1& &    \\
&    &      CART (no)& 4& 2& 3& 1& &    \\

&&      {\G}      K=5 & {\G}  & {\G} 3 & {\G} 6 & {\G} 1 & {\G} & \\
learners                    &&   {\G}       K=1 & {\G}  & {\G} 5 & {\G}  & {\G} 5     &{\G}\\
       &              &      {\G}   PCR &  {\G} & {\G} 4 & {\G}  & {\G} 1 & {\G} 5&\\
        &              &     {\G}   PLR & {\G} 4 & {\G}  & {\G} 4 & {\G}  & {\G} 2&\\

                     &  &     LR& & 1& & 3& 6&  \\
       &  &    SWR& & & 6& 2 &2&\\
                      &   &   NNet& & & 2& 2& 6&\\\cline{3-8}

      &          & SWR&2&4&1&2&\\
&             &   SFS& &7&1&1& \\

&& {\G}  none& {\G} 2& {\G} 3& {\G} 3& {\G} 1& {\G} &\\
pre-processors          && {\G} log& {\G} 2& {\G} 1& {\G} 1& {\G} 3& {\G} 2&\\
        && {\G}   norm& {\G} 2& {\G} 1& {\G} 2& {\G} 2& {\G} 2&\\
	   && {\G} PCA& {\G} & {\G} 4& {\G} 3& {\G} 1& {\G} 1& \\

     && freq5bin&&1&2&1&5&\\
&& width2bin&&&3&3&3&\\
    &&  width5bin&&&3&3&3&\\
    && freq3bin&&&1&3&5&\\
\cline{3-8}
\end{tabular}
}

\caption[Frequency of Methods in Ranks]{  Summary of position of methods 
}\label{table:freq_method}

}
\end{table*}

\subsection {Pred(25) Spectrum Analysis}
Figure \ref{fig:error_pred_datasets} shows the Pred(25) performances of all the datasets, sorted and grouped by their performance similarity into 4 classes, each class consists of 5 dataset members. 
All four groups performed consistently similar with different prediction techniques. Similarly Figure \ref{fig:error_pred_methods} evaluates the effectiveness of 90 different methods by grouping them into 5 categories. From the graph we observe that the Pred(25) performance of these 5 method categories are quite disperse and without overlapping, especially for the method group (71-90), which presents a linear trend. 



\begin{figure}
\begin{center} \includegraphics[width=3in]{lib/pred25sByData.pdf} \end{center}
\caption{Spectrum of Pred(25) across the datasets}\label{fig:error_pred_datasets}
\end{figure}

\begin{figure}
\begin{center} \includegraphics[width=3in]{lib/pred25s.pdf} \end{center}
\caption{Spectrum of Pred(25) down the methods}\label{fig:error_pred_methods}
\end{figure}

\subsection {Error Estimates}
Figure \ref {fig:error_methods} shows the total number of losses of each method, using all the datasets in the study. It clearly shows that there is a general conclusion that method 1-8 are performing consistently better than any other methods, method range 8-60 can maintain a similar number of losses for each method. Method range 60-90 performed badly. Method 1-8 belongs to the CART prediction family, consistenly outperformed other methods.

Figure \ref {fig:error_datasets} shows the total number of losses across the datasets, a much more stable condition exists if dataset china is to be removed. 


\begin{figure}
\begin{center} \includegraphics[width=3in]{lib/rows.pdf} \end{center}
\caption{Errors down the methods}\label{fig:error_methods}
\end{figure}


\begin{figure}
\begin{center} \includegraphics[width=3in]{lib/cols.pdf} \end{center}
\caption{Total number of losses across the datasets}\label{fig:error_datasets}
\end{figure}


\begin{figure}

\begin{center}
{\scriptsize
\begin{tabular}{r|rr|r}
  &\multicolumn{3}{c}{PRED(25)}\\\cline{2-4}
&norm&log& \\
data&CART(yes)&K=1& difference\\\hline
china&95&43&-52\\
sdr&42&17&-25\\
desharnaisL2&48&32&-16\\
nasa93\_center\_1&58&42&-16\\
cocomo81e&18&4&-14\\
desharnais&43&37&-6\\
nasa93\_center\_5&36&33&-3\\
maxwell&32&31&-1\\
albrecht&33&33&0\\
miyazaki94&40&40&0\\
desharnaisL1&39&41&2\\
cocomo81o&29&33&4\\
finnish&61&66&5\\
cocomo81&13&22&9\\
cocomo81s&9&18&9\\
nasa93&29&41&12\\
kemerer&7&20&13\\
nasa93\_center\_2&43&59&16\\
desharnaisL3&20&40&20
\end{tabular}}\end{center}

\caption{Comparing Pred(25) between ``best'' (at rank=1) and ``simplest'' (at rank=11)}\label{fig:error_pred_comp}
\end{figure}


\section{Discussion}
\input{jacky/validity}


\section{Conclusion}

This study empirically examined nearly one hundred different combinations of data preprocessors and algorithms executed on 20 software project datasets. 
For over half of the datasets executed, most cost estimation algorithms performed nearly the same, at least their differences are not statistically significant. 
The implication is that the traditional evaluation criteria such as MMRE may be less than important. 
There exists an ordering to the best effort estimator in our experiment, CART performed consistently better than any other algorithms throughout the execution of all the datasets, this is unexpected and not commonly known to the research community in software metrics.  

From all of these important findings, we can conclude that the features and quality of the datasets are far more important that features and quality of the algorithms. Without having investigation the characteristics and quality of the dataset being used in an experiment, the result produced may be flawed with influential biases inherited within the dataset. Our study also shows that a suitable data preprocessor such as stepwise feature selection will improve the overall prediction accuracy.  

We recommend researchers to explore the important research area of data quality rather than the space of possible algorithms which may produce a small improvement that is statistically insignificant. Data quality certification and data preprocessing processes are subject to further investigation, combining these techniques together with robust algorithms will lead to ultimate success in effort estimation. 
 










\bibliographystyle{abbrv}
\bibliography{icse, toAdd} 
% You must have a proper ".bib" file
%  and remember to run:
% latex bibtex latex latex
% to resolve all references
%
% ACM needs 'a single self-contained file'!
%
\appendix
\section*{Data}
\input{ekrem/aboutdata}

\end{document}
