Software effort estimation is, as its name suggests, the task of estimating the amount of effort required to develop new software.
It is essential for software developers to estimate a new project's effort in order to make a decision about developing or not developing a software project.
In large projects, it is critical that all resources are planned ahead of time and money is allocated to the project. In many cases, the success or failure 
of a company or organization may depend on such a software project.

According to Boehm, software effort estimates are often wrong by a factor of
four \cite{boehm81}. Other studies have reported larger errors (of 85\% to 772\% with many in the range of 500\% to 600\%)
when comparing different software models \cite{kemerer87}. A recent example is the NASA's CLCS system, which was cancelled in 2003. 
The initial estimate for the project was 206 million dollars, which was increased to between 488 million and 533 million dollars, more than twice
the initial estimate. On cancellation, approximately 400 developers lots their jobs \cite{clcs03}. 

There are other studies exploring the problem of overrunning projects. Molokken and Jorgensen report that 60\% to 80\%
of projects ``encounter effort and/or schedule overruns \cite{molo03}.'' According to Standish Group's Chaos Report, ``31.1\% of projects will be
cancelled before they ever get completed. Further results indicate 52.7\% of projects will cost 189\% of their original estimates \cite{standish95},''
meaning that there is a 89\% overrun in these projects.

This could mean that the remaining projects may have to be cancelled or postponed in order to cover the costs of these projects.
Although Jorgensen and Molokken briefly question such numbers \cite{jorgensen04}, they report an average of 30\% to 40\% overrun \cite{molo03,jorgensen04}.
There are other studies reporting several cancelled, delayed, and overrun projects. The lesson from these studies is to search for methods
that can reduce the error in effort estimation and produce reliable and accurate estimates.
 
Researchers have developed several software models and numerous effort estimation methods, some of which can be found in \cite{myrtveit05}.
This has caused two problems. One is that, each organization capable of collecting historical data has used a different model to collect data
and most likely kept it private. Therefore, there is usually no access to data in order to research certain models' performance. This has led
this study to use the COCOMO model developed and later improved by Boehm \cite{boehm81,boehm00b} with two publicly available data sets.

Another problem is that, although there are numerous studies on different effort estimation methods
(for example, Jorgensen and Shepperd's study \cite{jorgensen07} identified 304 software cost estimation papers),
there are few empirical studies comparing these methods, and more importantly, reporting stable conclusions.
There are, however, few studies that compare different software models, such as COCOMO II, SEER-SEM, PRICE-S comparison in \cite{lum02},
REVIC, SASET, PRICE-S, SEER-SEM, SLIM, SOFTCOST, CHECKPOINT, COCOMO II, and SAGE comparison in \cite{ferens98}, and
SLIM, COCOMO, Function Points, and ESTIMACS comparison in \cite{kemerer87}.
Although these studies are helpful in the sense that they provide a unique opportunity to compare the performance of these models (mostly without
any notable publicly available data), they also add to the confusion of what model or method to use since they either 
report no clear winner \cite{ferens98,lum02} or evaluate the results differently and offer different conclusions based on the method 
of evaluation \cite{kemerer87}.

It is clear that unless the numerous effort estimation methods are ranked systematically, the confusion will remain, and will possibly get worse
as new methods are added. This study aims at such a task. Different combinations of the representatives of several well-known groups of methods are
ranked in this research in an effort to introduce a framework for such rankings in the future, as well as reporting the best performing methods among 
the 312 combinations of methods studied here.

The importance of the results of this research is not only in the systematic method of comparison offered, but is
also in the stability of the conclusions made. The evaluation method used in this research plays a key role in illustrating this stability,
which is the main feature missing from many studies \cite{me06d,ferens98,lum02,kemerer87,shepperd01}.
As an example, Shepperd and Kadoda's study attempted at ``comparing four contrasting techniques of stepwise regression, rule induction, a form of
case-based reasoning, and neural nets \cite{shepperd01}.'' However, they were unable to show conclusive results about these methods and their
possible rankings relative to each other. It should be noted that their experiments used simulated data (first proposed by Pickard et al. \cite{pickard99}
where they explored simulated non-normal data) and were repeated across different data sets and used different sampled training sets in different runs.
Although no conclusive results were offered, their study contained some elements of a systematic comparison, such as different data sets and runs.

The experiments in this work have the same goal of ranking methods and use a similar approach. The COSEEKMO tool \cite{me06d} is further 
developed to incorporate several different methods, including model trees \cite{quinlan92b}, linear regression methods, case-based reasoning 
methods \cite{shepperd96,myrtveit05} (using estimation by analogy), thorough column pruning methods such as Wrapper \cite{kohavi97},
local calibration \cite{boehm81} methods, as well as newly developed column pruners \cite{baker07} and row pruners (Section \ref{s:locomo}).

There is an evaluation bias in every study of effort estimation methods, due to the selection of specific tests to compare these methods.
This study will introduce a non-parametric evaluation method called the Mann-Whitney U test (Section \ref{s:mwu}), which does not assume an underlying
distribution and hence, is less affected by the nature of the experiments. In order to further reduce the bias in evaluation, three
different evaluation criteria will be used to measure the performance of each method.
Also, different data sets (and their subset) and different random runs of the experiments will be used to compare all methods involved.

The results of this research show a great amount of stability in the conclusions (unlike other prior studies) and report four effort estimation methods
as the best performing methods among the 312 combinations of methods explored. These methods are all based on the local calibration method developed 
by Boehm \cite{boehm81}. A row pruning method developed in this work is used in two of these four methods.
These four methods were recently implemented in a tool called 2CEE for NASA's Jet Propulsion Laboratory and have shown outstanding results \cite{baker07}.

\section{Contributions of This Thesis}

This thesis has made the following contributions to both the literature and practice:

\begin{itemize}
\item A new framework for comparing and ranking effort estimation methods.
\item One of the few studies that compared the representatives of many effort estimation methods in a large scale, through 312 combinations of such methods,
and reported stable results.
\item Reducing the search space for effort estimation methods to a small group of four top performing methods.
\item Exploring the evaluation bias in the comparison of effort estimation methods and aiming at reducing its effects in such comparisons.
\item Suggesting the Mann-Whitney U test as a non-parametric evaluation method in effort estimation, which reduces the instability in the results.
This suggestion was recently taken into consideration by Baker \cite{baker07} and showed similar stability in the results.
\item A simple row pruning method, called Locomo and based on the local calibration procedure and the nearest neighbor algorithm,
which performs well and is reported in the top four performing methods in this study.
\item Industrial implementation of Locomo in the 2CEE tool developed for NASA's Jet Propulsion Laboratory.
\end{itemize}


\section{Structure of This Document}

The outline of the remaining chapters is as follows:

\begin{itemize}
\item Chapter 2 explores related studies and provides a background of the current research.
\item Chapter 3 gives the details of the experiments performed in this study and introduces tools and methods used or developed for this research.
It also offers the results and the discussion of these experiments.
\item Chapter 4 summarizes this study and offers conclusions. It also describes possible future works.
\item In addition, there is an appendix containing the instructions for the experimental tool used in this study.
\end{itemize}
