\documentclass{sig-alternate}
\usepackage{times}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{multirow}
\usepackage{rotating}


%\usepackage[none]{hyphenat} 
\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{itemize}
  \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{itemize}}

\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
   \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}


\usepackage[table]{xcolor}
\usepackage{url,graphicx}
\newcommand{\G}{\cellcolor[rgb]{0.8,0.8,0.8}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\hyp}[1]{Hypothesis~\ref{hyp:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}

\newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
\newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}

\begin{document}
%
% --- Author Metadata here ---
\conferenceinfo{ICSE}{2011 Waikiki, Honolulu, Hawaii USA}
\CopyrightYear{2011} % Allows default copyright year (20XX) to be over-ridden - IF NEED BE.
%\crdata{X-XXXXX-XX-X/XX/XX}  % Allows default copyright data (0-89791-88-6/97/05) to be over-ridden - IF NEED BE.
% --- End of Author Metadata ---

%\title{An Empirical Study on the Influence of Evaluation Criteria used in Software Cost Estimation  }
\title{Where is the Best Effort Estimator? }

%\numberofauthors{4}
%\author{
% You can go ahead and credit any number of authors here,
% e.g. one 'row of three' or two rows (consisting of one row of three
% and a second row of one, two or three).
%
% The command \alignauthor (no curly braces needed) should
% precede each author name, affiliation/snail-mail address and
% e-mail address. Additionally, tag each line of
% affiliation/address with \affaddr, and tag the
% e-mail address with \email.
%
% 1st. author
%\alignauthor
%Author1
% 2nd. author
%\alignauthor
%Author2
% 3rd. author
%\alignauthor 
%Author3
%\and  % use '\and' if you need 'another row' of author names
% 4th. author
%\alignauthor 
%Author4

%}


\numberofauthors{2} %  in this sample file, there are a *total*
% of EIGHT authors. SIX appear on the 'first-page' (for formatting
% reasons) and the remaining two appear in the \additionalauthors section.
%
\author{
% You can go ahead and credit any number of authors here,
% e.g. one 'row of three' or two rows (consisting of one row of three
% and a second row of one, two or three).
%
% The command \alignauthor (no curly braces needed) should
% precede each author name, affiliation/snail-mail address and
% e-mail address. Additionally, tag each line of
% affiliation/address with \affaddr, and tag the
% e-mail address with \email.
%
% 1st. author
\alignauthor Jacky W. Keung\\
    \affaddr{National ICT Australia Ltd. Australia}\\
    \affaddr{CSE, University of New South Wales, Australia}\\
    \email{Jacky.Keung@nicta.com.au}
\\
\alignauthor  Ekrem Kocaguneli, Tim Menzies\\
            \affaddr{CS \& EE, WVU, Morgantown, USA}\\
       \email{ekocagun@mix.wvu.edu, tim@menzies.us }
}



\maketitle
\begin{abstract}
\input{abstract}
\end{abstract}

% A category with the (minimum) three required fields
%\category{H.4}{Information Systems Applications}{Miscellaneous}
%A category including the fourth, optional field follows...
%\category{D.2.8}{Software Engineering}{Metrics}[complexity measures, performance measures]
\category{Software Engineering}{Software Metrics}{Data Mining}

\terms{Software effort Estimation, Linear Regression, Regression Trees, Neural Nets, Analogy, MMRE, Evaluation Criteria}

\keywords{Software Effort Estimation, MMRE, Evaluation Criteria}

\section{Introduction}

For decades, reserachers have been seeking the ``best'' software development effort estimator.
To date, no such ``best'' estimator has been found. This is an open and urgent
issue since accurate effort estimation is vital in software
engineering, and is often a challenging task for many software
project managers. Both overestimating and underestimating would
result unfavourable impacts to the business's competitiveness and project resource
planning. 
%
%The literature reports many  effort estimation methods.
%For example, if we just look at instance-based methods,
%\fig{cbr} lists thousands of  
%algorithms  for instance-based effort estimator. 
%A similiar list for model-based 
%effort estimation would be just as long
%(and include 
%linear regression, regression trees, neural nets, etc.).
%
%\input{ekrem/combination-algorithms}

Traditionally, researchers has focused  on the
{\em learner} used to generate the estimate (e.g. linear regression, neural nets, etc)
in many cases, overlooking the importance of the quality and characteristics of  the {\em data} being used
in the estimation process. This approach is somewhat misguided since learner
performance is controlled more by data preprocessing and the data sets used to   evalaute the learner.
This paper shows that many data sets used by prior publications are very {\em weak}
at distinguishing between different learners. All the results based on these {\em weak} data sets
(including many results by the authors of this papers) must hence be revisited.

The good news is that there are {\em strong} data sets that clearly illustrate the
value of a particular estimatior. 
Future effort estimation research should base its conclusions on these  {\em stronger}
data sets.

The rest of this paper is structured as follows.
Our releated work section notes that prior research has commented on the {\em conclusion instability}
problem in effort estimation (i.e. different reports offer different rankings to estimators). However,
those reports used single
data set to {\em seed} the generation of artifical data sets. Our results section shows that
if we experimented on a broad enough set of data, we can generate stable conclusions; e.g.
that some data sets are {\em weak}; and that the {\em strong} data sets
show that  some estimators are consistely better than others.
Based on those results, our conclusions will recommend (and depricate) particular estimators.
\input{ekrem/data}

\section{Related Work}

\input{jacky/related}
\input{ekrem/accuracy}

%\section{Hypothesis}







\section{Experiment Design}
    
As mentioned in the introduction, it is impractical to study {\em all} effort estimation methods
as there are too many different combinations. The following section describes the datasets, the algorithms and the evaluation method applied in the experiment.

\subsection{Datasets}

Figure \ref {fig:data} summarizes the datasets that were used for our comparison study using different data processors and learner algorithms. As can be seen from the list the datasets, they are quite diverse and were collected from many different software application domains and development environments across different continents in the world. A total of 11 publicly available datasets were used, and an additional 9 homogenised data subsets were extracted from COCOMO81, NASA93 and Desharnais datasets to examine the effects of homogeneity and its influence on the prediction results.  Altogether, 20 real industry real software project datasets were used in the study. It is important to stress that none of these datasets was selected in favour of a particular prediction approach, but we can see their widespread use in the effort estimation literature. All the datasets are available from the PROMISE repository. 

\subsection{Performance Measures}
Many performance criteria are used in the effort estimation literature including the six used in this study.

The absolute residual (AR) is the difference between the predicted and the actual:
{\begin{equation}\label{ar}
AR_i={x_i-\hat{x_i}}
\end{equation}}
(where $x_i,\hat{x_i}$ are the actual and predicted value for test instance $i$).

The Magnitude of Relative Error measure a.k.a. MRE is a very widely used evaluation criterion for selecting the best effort estimator from a number of competing software prediction models\cite{shepperd97}\cite{foss}. MRE measures the error ratio between the actual effort and the predicted effort and can be expressed as the following equation:  
{\begin{equation}\label{one}
MRE_i=\frac{\mid{x_i-\hat{x_i}}\mid}{{x_i}} = \frac{\mid{AR_i}\mid}{{x_i}}
\end{equation}}
A related measure is MER:
{\begin{equation}\label{one}
MER_i=\frac{\mid{x_i-\hat{x}_i}\mid}{\hat{x}_i} = \frac{\mid{AR_i}\mid}{\hat{x}_i}
\end{equation}}

The overall average error of MRE can be derived as the Mean or Median Magnitude of Relative Error measure (MMRE, or MdMRE respectively), can be calculated as: 

{\begin{eqnarray}\label{one}
MMRE={\sum_{i=1}^n   MRE_i\over {n} } \\
MdMRE={median(all MRE_i) }
\end{eqnarray}}

Foss et al.\cite{foss} have provided an extensive discussion using a simulation study demonstrating that the different characteristics of different variants of MREs to select the best prediction model, and their inherited weaknesses. Their recommendation was to always using a combination of theoretical justifications. For example, using MMRE together with a statistic test such as Student's {\em t}-test to formally verify the interpretation of MMRE value.


A common alternative to MMRE is PRED(25), and defined as the percentage of predictions failing within 25\% of the actual values, and can be expressed as:
\begin{equation}
PRED(25) = \frac{100}{N}\sum_{i=1}^N\left\{\begin{array}{ll}
\mbox{1 if $MRE_i \le \frac{25}{100}$}\\
\mbox{0 otherwise}
\end{array}\right.
\end{equation} 

For example, PRED(25)=50\% implies that half of the estimates are failing within 25\% of the actual values \cite{shepperd97}.

There are many other performance measures including the mean inverted balanced relative error (MIBRE) studied by Foss et al.~\cite{foss}:
\begin{equation}
MIBRE_i = \frac{\hat{x_i} - x_i}{max(\hat{x_i},x_i)}
\end{equation} 
Initially, we planned to explore more peformance measures than
AR, MRE, MER, MMRE, MdMRE, PRED(25) and MIBRE. However, as discussed below, all these measures lead to the same overall 
rankings of algorithms and data sets. Hence, exploring further measures seemed somewhat redundant.

\subsection{Wilconxon signed rank test}

The study applied the Wilcoxon signed rank test (95\% confidence) to summarize the results of the MRE and PRED(25) measures. Wilconxon is a non-parametric statistical hypothesis test commonly used to compare the difference between two groups of sample data, and it is used as an alternative to the paired Student's {\em t}-test when the population distribution cannot be assumed to be normal. It is more robust than the Student's {\em t}-test as it compares the sums of ranks, unlike Student's {\em t}-test which may introduce spurious findings as a result of presence of outliers may be existed in the given datasets. Ranked statistical tests like the Wilcoxon are also useful if it is not clear that the underlying
distributions are Gaussian~\cite{kleijnen97}. For example in a ranking analysis, the raw results 
(10.2, 21.3, 22.1, 24, 25, 30, 100) are
replaced with their ranks in the sort order; i.e. 
(1, 2, 3, 4, 5, 6, 7).


The pseudocode for win-tie-loss calculation is given in Figure \ref {FigureWinTieLossPseudocode}.
To calculate the win-tie-loss values, we first to check if two distributions $i, j$ are statistically different according to the Wilcoxon test. Otherwise we increment $tie_i$ and $tie_j$. On the other hand, if they returned to be different, we update $win_i,win_j$ and $loss_i,loss_j$ after a numerical comparison of their median values. The following experiment will draw results based on the Wilconxon test. 

\begin{figure}
\renewcommand{\baselinestretch}{1}
\begin{algorithmic} \footnotesize
\STATE \textit{$win_{i}$} = 0, \textit{$tie_{i}$} = 0, \textit{$loss_{i}$} = 0
\STATE \textit{$win_{j}$} = 0, \textit{$tie_{j}$} = 0, \textit{$loss_{j}$} = 0
\IF{WILCOXON(\textit{$P_i$}, \textit{$P_j$}) says they are the same}
	\STATE \textit{$tie_{i}$} = \textit{$tie_{i}$} + 1;
	\STATE \textit{$tie_{j}$} = \textit{$tie_{j}$} + 1;
\ELSE
	\IF{median(\textit{$P_{i}$}) $<$ median(\textit{$P_{j}$})}
		\STATE \textit{$win_{i}$} = \textit{$win_{i}$} + 1
		\STATE \textit{$loss_{j}$} = \textit{$loss_{j}$} + 1
	\ELSE
		\STATE \textit{$win_{j}$} = \textit{$win_{j}$} + 1
		\STATE \textit{$loss_{i}$} = \textit{$loss_{i}$} + 1
	\ENDIF
\ENDIF
\end{algorithmic}
\caption{Pseudocode for Win-Tie-Loss Calculation Between Variant \textit{i} and \textit{j}
with performance measures $P_i$ and $P_j$.}
\label{FigureWinTieLossPseudocode}
\end{figure}





\input {ekrem/pre-processor}
\input {ekrem/learner}

\section{Results}
Based on our experimental setting, a total of 1,800 estimation results (20 datasets $\times$ 9 preprocessors $\times$ 10 learners) were generated, they were then ranked according to the number of losses using the Wilcoxon test (Figure \ref {FigureWinTieLossPseudocode}), and classified into 4 different groups of percentage of losses, illustrated by using 4 different colours in Figure \ref{fig:dot-plot}. 
The sum of the losses seen in a leave-one-out experiment using
AR, MRE, MER, MMRE, MdMRE, PRED(25), or MIBRE are shown in
Figure \ref{fig:dot-plot}. 
Each dot in the figure represents 623 compariosons
(each algorithm compared to 89 others using seven performance measures).
The 
x-axis of that figure shows the various data sets
and the 
y-axis of that figure shows the 90 
algorithms. For space reasons, the names of algorithms are shown in Table~\ref{table:res_rank}.
For example, the algorithm at rank 1 (that is, with fewest total losses) is norm-CART(yes); i.e.
the CART learner with all the numerics normalized min..max as 0..1; and with tree post-pruning enabled.




\begin{figure}[!t]
\begin{center}
\includegraphics[width=3in]{lib/thinks.pdf} \end{center}
\caption[Dot-Plot]{Overall estimation results using different methods, ranked by the number of losses.} \label{fig:dot-plot}
\end{figure}

In the leave-one-out results of 
Figure~\ref{fig:dot-plot},
the algorithms on the y-axis are ordered by the total number of losses across all peformance measures in each row.
Further,  the data sets on the x-axis are ordered by the total  
number of losses in each column. This ordering is nearly the same as we view:
\bi
\item The number of the losses for each individual performance measure;
\item The number of losses seen in the 10$\times$3-way cross-validation results;
\item  Or if we view the number of wins.
\ei
Here, by ``nearly the same'', we mean that the rank position of a particualr dataset/algorithm never changes by more than 10\%.
Hence, for space reasons, we just show
Figure~\ref{fig:dot-plot},


\subsection{The World is ``Flat''}

We have observed over 50\% of the estimates have Wilcoxon losses less than 12.5\%, only a small fraction of the results with losses greater than 50\%.
In all 90 method combinations (Table \ref {table:res_rank}), CART ({\em Classification And Regression Tree}) had the least losses, that is, CART consistently achieves lower MRE values of all the methods using given datasets in the study. For top 10 method combinations, CART as a learner outperformed all other prediction models, such as Analogy and Linear Regression Analysis. 

In addition to the given results shown in Table \ref{table:freq_method}, there are some interesting findings deserve some attention. Firstly, as mentioned above, CART performed consistently better than any other methods as a learner, this is evident in the learner section of Table \ref{table:freq_method}, where CART (subtree search) and CART (full tree search) ranked in the top according to the least losses in the race. Interestingly, the worst learners are the Regression methods (SWR, and LR) and the NeuralNet method (NNet).The best combinations is (simple max/min data normalization with CART). 

Although SWR (stepwise regression) as a learner did not perform well, but as a pre-processor in the prediction system it is significantly important. Of the 90 different combinations, SWR leads to least losses combining with different learners. This is evident by its occurrence indicated in Table \ref{table:freq_method} showing a summary of position of methods, 2 SWRs exists in the top ranked category, and 4 exists in the 2nd top ranked category.   The 2nd most important data pre-processor is SFS (Stepwise Forward Selection). The implication in here is that combining SWR or SFS with any other learners will result in favourable prediction accuracy, as both stepwise procedures (SWR and SFS) are likely to statistically filter only the relevant project features for the learners to predict estimates. The finding coincides with Keung et al.'s work \cite{keung08a}, which argues the importance of a data preprocessor in software effort estimation, in their experiments, a stepwise feature selection system similar to that of SWR was adopted, results showing improved prediction performance as well. 

Finally, a very important finding was found relating to different evaluation criteria. Our results show that regardless of the evaluation criteria used (AR, MRE, MMRE, MdMRE, PRED25 and MIMRE) they all lead to the same overall rankings of algorithms and datasets. 
The implication is that different evaluation criteria can be used, as using different evaluation criteria are statistically insignificant. This perhaps leads to the end of a continuous debate as to whether we should use MMRE, MdMRE, etc. or an alternative metric for comparing different prediction models. 

From the dot-plot graph we can see that the contour data pattern showing that the "world is flat", especially based on the first few datasets such as telecom, kemerer, cocomo81o. The implication is that using different prediction models may result similar findings in most cases, again this is largely depends upon on the dataset characteristics and quality. In this study, dataset China is a cross-company dataset, performed worst among the 20 datasets, nevertheless, CART learner is still the preferred method which result in least losses. 

Given similar estimates can be achieved using different methods, we are arguing that using a simpler solution may be adequate in most of the cases. 
To support our claim, we compared ({\em simple max/min data normalization with CART}) with ({\em simple log norm.+ KNN=1}) measured by Pred(25) in Figure  \ref{fig:error_pred_comp}. As we have observed, except china and sdr datasets, the other differences are not significant. In particular, the two method combinations using albrecht and mizayaski94 datasets produced identical estimates.



\input{ekrem/algorithm-ranks}

\input{ekrem/position-summary}

\subsection {Spectrum Analysis}

Figure \ref{fig:dot-plot} shows the overall estimation results based on the 20 datasets, these datasets are sorted by the total number of losses with different methods and shown in the x-axis. To investigate the distribution from a different perspective, we divide all the datasets used in the experiments into 4 groups, and each group consists of 5 datasets. Figure \ref{fig:error_pred_datasets} shows the estimation spectrum across all the datasets based on Pred(25), and a similar data pattern was formed in all 4 groups. 

Similarly Figure \ref{fig:error_pred_methods} evaluates the effectiveness of 90 different method combinations by grouping them into 5 categories. Method combination 1 to 8 performed the best among all 5 categories and a linear trend is present in each category. 


\begin{figure}
\begin{center} \includegraphics[width=3in]{lib/pred25sByData.pdf} \end{center}
\caption{Spectrum of Pred(25) across the datasets}\label{fig:error_pred_datasets}
\end{figure}

\begin{figure}
\begin{center} \includegraphics[width=3in]{lib/pred25s.pdf} \end{center}
\caption{Spectrum of Pred(25) down the methods}\label{fig:error_pred_methods}
\end{figure}

\subsection {Error Estimates}
Figure \ref {fig:error_methods} shows the total number of losses of each method, using all the datasets in the study. It clearly shows that there is a general conclusion that method 1-8 are performing consistently better than any other methods, method range 8-60 can maintain a similar number of losses for each method. Method range 60-90 performed badly. Method 1-8 belongs to the CART prediction family, consistently outperformed other methods.

Figure \ref {fig:error_datasets} shows the total number of losses across the datasets. 
The subtle increase in the graph was caused by the inclusion of the china dataset. 
This suggesting some of the datasets should not be used for effort estimation, we recommend to avoid datasets with the number of losses > 10,000 cases, including dataset china, sdr, maxwell etc. as shown in Figure \ref {fig:error_datasets} .


\begin{figure}
\begin{center} \includegraphics[width=3in]{lib/rows.pdf} \end{center}
\caption{Errors down the methods}\label{fig:error_methods}
\end{figure}


\begin{figure}
\begin{center} \includegraphics[width=3in]{lib/cols.pdf} \end{center}
\caption{Total number of losses across the datasets}\label{fig:error_datasets}
\end{figure}


\begin{figure}[!t]

\begin{center}
{\scriptsize
\begin{tabular}{r|rr|r}
  &\multicolumn{3}{c}{PRED(25)}\\\cline{2-4}
&norm&log& \\
data&CART(yes)&K=1& difference\\\hline
china&95&43&-52\\
sdr&42&17&-25\\
desharnaisL2&48&32&-16\\
nasa93\_center\_1&58&42&-16\\
cocomo81e&18&4&-14\\
desharnais&43&37&-6\\
nasa93\_center\_5&36&33&-3\\
maxwell&32&31&-1\\
albrecht&33&33&0\\
miyazaki94&40&40&0\\
desharnaisL1&39&41&2\\
cocomo81o&29&33&4\\
finnish&61&66&5\\
cocomo81&13&22&9\\
cocomo81s&9&18&9\\
nasa93&29&41&12\\
kemerer&7&20&13\\
nasa93\_center\_2&43&59&16\\
desharnaisL3&20&40&20
\end{tabular}}\end{center}

\caption{Comparing Pred(25) between ``best'' (at rank=1) and ``simplest'' (at rank=11)}\label{fig:error_pred_comp}
\end{figure}


\section{Discussion}

\input{ekrem/data-in-literature}

\input{jacky/validity}


\section{Conclusion}
In this study, ten learners and nine data preprocessors were combined
 into 90 effort estimation algorithms. These
were applied to twenty data sets using two experimental rigs:  a
leave-one-out study; and  a 10*3-way cross-val study.  Performance
was measured using  seven performance indicators (AR, MRE, MER,
MdMRE, MMRE, PRED(25), MBIRE). Performances were compared using a
Wilcoxon ranked test (95%).

The following results were stable across the seven performance
indicators and the two experimental rigs.
Our  data sets   could be sorted according to how well
they can distinguish between  effort estimators; specifically, ten
data sets (in common use  in the effort estimation literature)   are
{\em weak}; i.e. poorly distinguish the behavior of different
estimators.  
The effectiveness of a learner used for effort
estimation (e.g. regression or analogy methods) can be significantly
altered by data pre-preocessing (e.g. logging all numbers or
normalizing them zero to one).  Stepwise regression was a very
useful preprocessor, but surprisngly poor learner.  Neural nets
and simple linear regression perform much worse than other learners
such as analogy learners.  Non-simple regression methods such
as regression trees and partial linear regression are relatively
strong performers.   Regression trees performed best of all in
this study (with a preprocessor that normalized the numerics zero
to one).    Very simple methods (e.g. K=1 nearest neighbor on
the log of the numerics)  performed nearly as well as regression
trees.   

Based on these results, we conclude that
prior reports of conclusion instability about effort
estimation may have been overly pessimistic. Given the large number
of publicly  available effort estimation data sets, it is now
possible to make stable conclusions about effort estimation. Specifically, we say that  
prior papers on effort estimation have not taken proper account of
the data used to evaluate the effort estimator.  Future publications
in effort estimation should take greater care to explore the
preprocessing, and not just the learner.
Also, researchers need to revisit prior publications
that used the   weaker data sets (including numerous papers by the
authors of this paper).  

Finally, this study recommends effort estimation using
regression trees or analogy methods. It also recommends against
effort  estimation based on neural nets or simple linear regression.
These recommendations are based only on the algorithms studied in this paper and
should be assessed using other algorithms.
That assessment should be based on the 
non-weak data sets identified by this study.  


%This study empirically examined nearly one hundred different combinations of data preprocessors and algorithms executed on 20 software project datasets. 
%For over half of the datasets executed, most cost estimation algorithms performed nearly the same, at least their differences are not statistically significant. 
%The implication is that the traditional evaluation criteria such as MMRE may be less than important. 
%There exists an ordering to the best effort estimator in our experiment, CART performed consistently better than any other algorithms throughout the execution of all the datasets, this is unexpected and not commonly known to the research community in software metrics.  
%
%From all of these important findings, we can conclude that the features and quality of the datasets are far more important that features and quality of the algorithms. Without having investigation the characteristics and quality of the dataset being used in an experiment, the result produced may be flawed with influential biases inherited within the dataset. Our study also shows that a suitable data preprocessor such as stepwise feature selection will improve the overall prediction accuracy.  
%
%We recommend researchers to explore the important research area of data quality rather than the space of possible algorithms which may produce a small improvement that is statistically insignificant. Data quality certification and data preprocessing processes are subject to further investigation, combining these techniques together with robust algorithms will lead to ultimate success in effort estimation. 
% 
%








\bibliographystyle{abbrv}
\bibliography{icse} 
% You must have a proper ".bib" file
%  and remember to run:
% latex bibtex latex latex
% to resolve all references
%
% ACM needs 'a single self-contained file'!
%
\appendix
\section*{Data}
\input{ekrem/aboutdata}

\end{document}


In order to support this claim,  this paper demonstrates a study of nearly one
hundred different algorithms executed on numerous data sets. For example,
this paper explores The experiment explores the following research questions:

{\bf RQ1:} Is there any stability in the conclusions gained from the currently available
data sets of real project data?

{\bf RQ2:} 
Our reading of the effort estimation
literature is that there is {\em little,little,extensive} focus on the {\em datasets,preprocessors,learners}.
Is this current emphasis correct and sound? 

{\bf RQ3:} Are there datasets to avoid when conducting estimation experiments?

{\bf RQ4:} Can we reasonably to declare a best effort estimator?


