This work had two main goals. One was to study the evaluation bias in effort estimation and the other 
was to report the relative performance of numerous methods studied in the literature and find the best performing methods.

\subsubsection{Studying Evaluation Bias In Effort Estimation}

One of the most important biases in evaluating effort estimation methods is the type of evaluation methods used.
There were several experiments with COSEEKMO comparing effort estimation methods using parametric evaluation.
Those experiments illustrated how parametric evaluation methods, applied as the standard tests in the literature, 
were unsuitable for effort estimation experiments in general, due to their normality assumptions.
As a result, this study introduced non-parametric tests as the new approach to evaluating effort estimation methods, and namely the Mann-Whitney U test.
The introduction of the MWU test was beneficial in two ways:

\begin{itemize}
\item As a non-parametric test, it could be applied to any distribution, including the error distributions in this experiment.
\item It showed a great amount of stability in comparison of different effort estimation methods, regardless of the evaluation criteria used.
This stability did not exist previously (in prior experiments as well the literature) when parametric tests were used (Section \ref{s:instability}).
\end{itemize}

Three different evaluation criteria, AR, MER, and MRE, were used in this study. The stability of the observations made across 
different evaluation criteria was clear. Section \ref{s:results} demonstrates how evaluation bias can be reduced using a non-parametric 
test such as the MWU test.
Furthermore, applying such a test illustrated stability beyond evaluation criteria. The stability of the observations
(Section \ref{s:resultsSimplerTop}) was also clear across different data sets and their subsets and across different random samplings of these
data sets through several random runs of the experiment. (Overall, 312 combinations of methods were applied to 2 data sets with 19 subsets through
5 random runs and finally evaluated using 3 evaluation criteria.) This is contrary to what Shepperd and Kadoda \cite{shepperd01},
among other researchers \cite{myrtveit05,kitc01,foss05}, reported regarding stability.

Nonetheless, such a bias cannot be completely removed, as shown in the last findings of Section \ref{s:resultsSimplerTop}. 
The space of explored evaluation biases in this study includes some of the well-known biases. Nonetheless, there may be
other evaluation biases introduced that could agree or disagree with the results of this study. Such biases can be explored in future works.
 
Unavoidably, due to the nature of this study, four other biases were also introduced.
These biases can affect each other, and hence, affect the evaluation bias. Thus, they are worthy of a brief discussion.
The first bias was in the paradigm. This study only explored the model-based methods, and among which, only the COCOMO 81 model.
On the other hand, expert-based methods, which require human expertise, could not possibly be explored in a fully automatic tool such as COSEEKMO.
However, a future study can be devised to compare the performance of model-based methods included in automatic tools such as COSEEKMO to those
of the experts.

This study only explored the COCOMO 81 model, introducing the second bias, which is the bias in the model. 
However, this could not be avoided since COCOMO 81 was the only well-known software model with publicly available data.
In addition, prior research using this model is extensively available, which makes it possible to measure the validity of the experiments
in this study to the ones in the literature. Nonetheless, this type of bias can be explored in future works, depending on the availability of more data.

The third bias is in the selection of methods used within the model-based effort estimation methods. There are many other methods in the
literature and practice that may produce different results. Although this study selected a number of methods representing several other methods, it is
worthwhile to explore other methods in future works and possibly further prune the space of effort estimation methods.

The fourth and last bias involves the sampling of the data, given the fact that historical data about the projects can be maintained in a given organization.
The data used in this study comes from (the only) two publicly available data sets collected by Boehm \cite{boehm81} and NASA (Section \ref{s:data}).
As a result, one may argue that using only two data sets introduces the sampling bias to the observations made.
However, it should be noted that these two data sets were repositories that accepted data from a wide range of projects.
For example, the $NASA93$ data set comes from different teams working at geographical locations spread
throughout the United States using a variety of programming languages. While some of this data is from flight systems (a particular NASA specialty),
most are ground systems and share many of the properties of other terrestrial software (such as same operating systems, development languages, 
development practices). Much of NASA's software is written by contractors who service a wide range of clients (not just NASA).
These contractors are contractually obliged (ISO-9001) to demonstrate their understanding and usage of current industrial best practices. 

Nonetheless, this study explored different subsets of these data sets in order to reduce the sampling bias. Increasing the number of random runs of the
experiments along with sampling at random to create different train and test sets were the measures taken along this path.
Similar to other biases, this bias can be explored in future works depending on the availability of more data.

\subsubsection{Studying Effort Estimation Methods}

The comparison of effort estimation methods in this research to study evaluation bias also made it possible to explore their relative performance
and report a group of methods as superior to others. Some of these methods including linear regression, model trees, and nearest neighbor (based on case-based
reasoning) were studied before. Some, such as the Cocomin column pruner, were recently developed \cite{baker07} and some, such as the Locomo row pruner, were
developed in this study. Out of 312 combinations of methods possible in COSEEKMO, a group of 8 representatives of the above methods were selected
through detailed analysis of the results in Section \ref{s:results}. This analysis made it possible to draw conclusions about the performance
of these methods, which helps in simplifying the effort estimation process by limiting the scope of the search for such methods to a 
small list of four methods. This leads to another observation stating that there is no single best effort estimation method.
Rather, there is a group of four methods that should be tried on the historical data and the best performing one can be applied to generate
effort estimates for a future project.

There were several observations made throughout Section \ref{s:results} that are organized here. These conclusions are applicable
to both $COC81$ and $NASA93$ data sets and their subsets as well different evaluation criteria and different runs.

\begin{itemize}
\item Simple methods based on local calibration perform better than other methods.
\item Applying one or both simple row pruning, using dynamic Locomo, and simple column pruning, using Cocomin, 
increases the performance of the local calibration learner.
\item Sophisticated and thorough column pruning (using Wrappers with target learners such as local calibration, LSR, or M5p) 
does not increase the performance of a method and always performs worse than Cocomin.
\item Row pruning without applying a learner (in the case of the nearest neighbor method that uses case-based reasoning) produces the worst
performance among all methods in this study. Hence, applying the domain knowledge is essential and therefore, similar case-based reasoning methods 
found in the literature are not as useful as the ones (such as Locomo) that employ local calibration. (According to \cite{myrtveit05}, case-based
reasoning methods belong to both sparse-data methods (requiring few or no historical data) and many-data methods (where historical data is
available). Therefore, the comparison of a case-based reasoning method with other methods in this study is justified. Moreover, the same
algorithm from a case-based reasoning method augmented with local calibration creates Locomo that performs well.)
\item Model trees fail to increase the performance of a method in effort estimation, emphasizing the assumption of local calibration
that effort is a single linear function. 
\item The methods that log the data (hence using linear regression) outperform the methods that do not log the data. This is one of the assumptions
used in local calibration, where effort is modeled as an exponential function on the lines of code (and is linearly proportional to the
product of a set of effort multipliers).
\item Using proximal numeric estimation failed to increase the performance of the methods in this study. This is another assumption used
in local calibration, where there is a pre-defined set of (precise) numeric values used as effort multipliers influencing the effort.
\item Applying the superset train type is not useful. Methods using the superset train type usually perform no better (and generally worse) than
methods using the subset train type. This could be due to the presence of more noise in the train set as it gets larger through applying
the superset train type rather than the subset train type, hence reducing the performance of the trained model.
As a result of this and since training using the superset takes more time, there is no reason to use the superset train type.
\end{itemize}

As a result of these observations, in order to obtain the best performance in $COC81$ and $NASA93$ data sets,
this study suggests using dynamic Locomo as a row pruner and Cocomin as a column pruner to be applied as pre-processors to local calibration.
Locomo and Cocomin are fast and use simple algorithms to search for best projects and features of a data set respectively.
There are a total of four combinations of these methods (no column or row pruning, only column pruning, only row pruning, and both column and row pruning)
with local calibration always used as the learner. Since these are fast methods and there is no single best method among these, 
this study suggest applying all four combinations to the historical data and selecting the one with the best performance to generate 
effort estimates on future projects.

Although Locomo and Cocomin are newly developed methods, they were tested very recently and similar findings to the results of this
study were reported. According to \cite{baker07}, when applied in the NASA's Jet Propulsion Laboratory environment,
``Locomo and Cocomin when used to estimate flight projects produced a 5.4\% median MRE, whereas standard LC produced a 21.97\%
median MRE, and SCAT provided a 29.29\% median MRE.'' This emphasizes the validity of the findings in Section \ref{s:resultsSimplerTop}, where
Locomo and Cocomin were found to improve the performance of local calibration. Similarly, ``using Locomo to estimate ground-based projects resulted 
in a 15.9\% median MRE, whereas standard LC and uncalibrated models such as SCAT produced 28.9\% and 42.1\% median MRE's, respectively \cite{baker07}.''
Locomo is once again found to provide better performance comparing to local calibration without any row pruning.

As mentioned before, (simple) local calibration methods reported the best performance among all other methods when
applied to the COCOMO 81 data sets. This is an important finding since it suggests that a data set is best explored using the models developed
for it rather than more generic, and possibly sophisticated methods such as model trees, or more simplified methods such as the nearest neighbor method.
Although this might seem obvious, there is a number of studies in the literature that ignore this fact. This study suggests applying or developing 
new and simple methods (such as Locomo and Cocomin) around the basic modeling tools (such as local calibration) developed for a software 
model (such as COCOMO 81) before creating more sophisticated ones that lack the domain knowledge about the data.

At the time of writing the results of this research, there is no known study that reports the stability seen in the results of the comparison of such a large
number of effort estimation methods. One reason may be the fact that such a comparison is not a widespread practice due to many factors such as the lack of data,
appropriate top-performing methods, and even resources. This study took more than a year and required hours of processing using computers as well as 
human analysis of the results. Furthermore, it required developing new methods and interacting with other researchers through this process. Therefore,
many studies prefer applying one or two and usually fewer than a handful of methods and report their results based on those findings.

A second reason is the approach to comparing and evaluating the performance of effort estimation methods. As an example, even when a relatively large number of
methods were compared in \cite{me06d}, the statistical methods applied to the results were wrong (since they used parametric tests, which this study proves 
them to be unsuitable). As a result, that study reported large standard deviations with $\{median,max\} = \{150\%,649\%\}$ and identified
the root of such problems to be the rare presence of large errors in the results that can mislead mean calculations. Hence, better evaluation methods
(such as rejection rules in Figure \ref{figure:rejectionRules}) were devised to reduce such deviations. However, little progress was made to report 
stability in the results \cite{me06d}. Other studies, such as Shepperd and Kadoda's study \cite{shepperd01}, report instability (or no stability)
in their results as well.

There may be other reasons for the absence of studies reporting stability, which is considered a source of contradicting studies in the effort 
estimation literature. Overall, the conclusions made in studies investigating the conclusion instability problem \cite{myrtveit05,shepperd01,kitc01,foss05},
which report doubts in the possibility of finding a solution to the instability problem (discussed in Section \ref{b:instability})
were shown not to hold in this study. This study offers stability, and more importantly a framework for comparing new methods, as well 
as other existing methods not in COSEEKMO, with the current list of best methods reported from this work.

\section{Future Work}
\label{c:future}

There are several aspects of this research that can be further explored.

\begin{itemize}
\item This study only focuses on model-based methods. Expert-based methods \cite{jorg04} 
are also important to consider when comparing the performance of methods.
The comparison of these two groups can only be done if experts are willing to take part in this experiment.
Fortunately, the number of top-performing effort estimation methods (among the ones studied) was reduced to four in this research.
Therefore, the length of such an experiment is decreased greatly, making it possible to compare expert-based methods with the four model-based methods. 
\item COCOMO is the only well-known and well-published software model with publicly available data. Therefore, this study only focused on COCOMO. However,
there is a large number of software models that can be explored if data is available. It is suggested that COSEEKMO or a similar tool is developed for
such comparisons.
\item Even within COCOMO, there is no access to certain data sets. If such data sets became available, further sampling and more extensive studies
can be done.
\item Evaluation bias cannot possibly be explored in a single study due to the large number of evaluation methods and criteria available. Although
this study explored well-known parametric tests an introduced a non-parametric test, other tests \cite{demsar06} and evaluation criteria 
can be studied in order to verify or correct the results of this study.
\item The number of methods explored in this study is large. However, there are certainly other effort estimation methods to be explored. As it is seen
in Figure \ref{figure:sparse-many}, COSEEKMO only explored a small portion of that space. Future works can explore this even further and include
methods based on neural nets or genetic algorithms. However, it is essential to study these methods systematically and offer stable 
conclusions or reasons for instability.
\end{itemize}
