\documentclass[12pt]{report}
\topmargin=0in
\headheight=0in
\headsep=0in
\oddsidemargin=0in
\evensidemargin=0in
\textwidth=6.5in
\textheight=9in

\usepackage{setspace}
\usepackage{graphicx}
\usepackage{algorithmic,cite,footmisc,multirow,url,amsmath}
\usepackage{color}
\usepackage{pslatex}
\usepackage{fullpage}
\usepackage{alltt}
\usepackage{rotating}
\usepackage{float}
\usepackage{pifont}
\usepackage{wasysym}

\singlespacing

\begin{document}
\thispagestyle{empty}

\title{\large Evaluation Bias in Effort Estimation}
\author{\normalsize Omid Jalali \\ \\ 
\normalsize Thesis submitted to the \\
\normalsize College of Engineering and Mineral Resources \\
\normalsize at West Virginia University \\
\normalsize in partial fulfillment of the requirements \\
\normalsize for the degree of \\ \\
\normalsize Master of Science \\
\normalsize in \\
\normalsize Computer Science \\ \\
\normalsize Tim Menzies, Ph.D., Chair \\
\normalsize Afzel Noore, Ph.D. \\
\normalsize Jairus Hihn, Ph.D. \\ \\
\normalsize Lane Department of Computer Science and Electrical Engineering \\ \\
\normalsize Morgantown, West Virginia \\
\normalsize 2008 \\ \\
\normalsize Keywords: COCOMO, effort estimation, row pruning, evaluation, Mann-Whitney U test\\ \\
\normalsize Copyright \copyright~2008 Omid Jalali 
}

\date{}
\maketitle

\begin{abstract}
\begin{center}
Evaluation Bias in Effort Estimation \vspace{1.5ex} \\
Omid Jalali
\end{center}
There exists a large number of software effort estimation methods in the literature and the
space of possibilities \cite{myrtveit05} is yet to be fully explored. There is little conclusive
evidence about the relative performance of such methods and many studies suffer from instability in
their conclusions. As a result, the effort estimation literature lacks a stable ranking of such methods.

This research aims at providing a stable ranking of a large number of methods using data sets based on COCOMO features.
For this task, the COSEEKMO tool \cite{me06d} was further developed into a benchmarking tool and
several well-known effort estimation methods, including model trees, linear regression methods, local calibration, and several
newly developed methods were used in COSEEKMO for a thorough comparison. The problem of instability was further
explored and the evaluation method used was identified as the cause of instability. Therefore, the existing evaluation bias
was corrected through a new evaluation approach, which was non-parametric. The Mann-Whitney U test \cite{mann47} is the non-parametric test
used in this study, which introduced a great amount of stability in the results.
Several evaluation criteria were tested in order to analyze their possible effects on the observed stability.

The conclusions made in this study were stable across different evaluation criteria, different data sets, and different
random runs. As a result, a group of four methods were selected as the best effort estimation methods among the explored 312 combinations of methods.
These four methods were all based on the local calibration procedure proposed by Boehm \cite{boehm81}. Furthermore, these methods
were simpler and more effective than many other complex methods including the Wrapper \cite{kohavi97} and model trees \cite{quinlan92b},
which are well-known methods in the literature.

Therefore, while there exists no single universal best method for effort estimation, this study suggests applying the four methods
reported here to the historical data and using the best performing method among these four to estimate the effort for future projects.
In addition, this study provides a path for comparing other existing or new effort estimation methods with the currently explored methods.
This path involves a systematic comparison of the performance of each method against all other methods, including the methods
studied in this work, through a benchmarking tool such as COSEEKMO, and using the non-parametric Mann-Whitney U test.

\end{abstract}

\doublespacing

\chapter*{Dedication}
\renewcommand{\thepage}{\roman{page}}
\setcounter{page}{3}

\begin{center}
\textit{To My Family} \\
\end{center}

\chapter*{Acknowledgments}
I would like to thank my advisor, Dr. Tim Menzies, for introducing me to software engineering research,
teaching me a great deal about data mining and effort estimation, and introducing me to outstanding researchers at 
the ICSE conference, allowing me to learn more than possible. He gave me the motivation to start this research, and despite 
his extremely busy schedule, has kept me motivated all along. I am and will always be grateful for everything he has done for me.

I am thankful to the Lane Department and West Virginia University. I have had the unique opportunity of learning
about computer science in this department and about many other branches of science at West Virginia University.
I would like to thank Dr. John Atkins for his thorough courses in databases and his help as my academic advisor,
Dr. Elaine Eschen for the great theory courses that made me more interested every day and for her patience in answering
my numerous questions, Dr. Tim Menzies for his data mining course that taught me invaluable things about this subject,
and Dr. Jim Mooney for his great operating system courses.

I would like to thank Dr. Jairus Hihn from NASA's JPL, Dr. Afzel Noore, and Dr. Tim Menzies for serving on my committee. I would like to thank
Dr. Hihn for his support and statistical advices. I would like to thank Karen Lum from JPL and Dr. Hihn for collaborating on our papers.

I want to thank Dan Baker, a great friend and colleague, who wrote Cocomin and provided a unique opportunity to test this work at JPL.
I also thank my friends Zach Milton, Brian Sowers, Nathan Moore, Phillip Green, Ous Elrawas, and DJ Boland for their help, support, and humor.

Finally, I thank my parents and my family for their constant support throughout my education.

\singlespacing

\tableofcontents
\listoffigures
\listoftables
\pagebreak

\doublespacing

\chapter{Introduction}
\setcounter{page}{1}
\pagenumbering{arabic}
\label{introduction}
\input{introduction}

\chapter{Background and Related Work}
\label{background}
\input{background}

\chapter{Laboratory Studies}
\label{studies}
\input{studies}

\chapter{Conclusions}
\label{conclusions}
\input{conclusions}

\appendix

\doublespacing

\chapter{COSEEKMO's Manual}
\label{a:manual}
\input{manual}

\singlespacing

\bibliography{refs,myrefs}
\bibliographystyle{plain}

\pagebreak
\thispagestyle{empty}

\end{document}
