\section{Effort Estimation Methods}
\label{sect:methods}

The effort estimation methods studied in this paper fall into two groups: {\em solo}-methods
and {\em multi-}methods.
{\em Solo}-methods are some combination of a {\em pre-processor} and a {\em learner}.
For example, Boehm's preferred effort estimation method uses a log transform as the pre-processor,  then linear
regression as the learner.
{\em Multi-}methods are combinations of solo-methods.
\subsection{Multi-Methods}
%Effort estimation literature has covered an extensive amount of algorithms and models.
%Models in effort estimation can be grouped as elaboration of single algorithms (i.e. finding the optimum parameters) or combination of these algorithms with various pre-processors.
%We refer to such single algorithm-based models as \textit{solo-methods}.
%We define a \textit{solo-method} as the combination of a pre-processor with an algorithm.
%In the previous section we have defined $9$ algorithms and $10$ pre-processors.
%We combine these two sets to form \textit{9 algorithms X 10 pre-processors = 90 solo-methods}.
%In this paper whenever we refer to a \textit{solo-method}, we are referring to a pre-processor and an algorithm combination.

%\subsection{Multi-Methods}

\textit{Multi-methods} combine two or more solo-methods.
%The question when we refer to combining of solo-methods is the combination strategy.
 Many combination schemes have been proposed in the literature~\cite{Kittler1998, Ali1995, Alpaydin1998, Mendes2003}.
Complex combination schemes include bagging~\cite{Breiman1996}, boosting~\cite{Freund1999} or random forests~\cite{Jiang2008a, Breiman2001}.
Simpler methods include
computing the
mean, median or inverse-ranked weighted mean (IRWM~\cite{Mendes2003}, see \fig{irwm}) of estimates coming from \textit{n-many} solo-methods.

Our aim is not to investigate complex schemes, but to observe how multi-methods perform compared to solo-methods on effort datasets.
Therefore, we adopt simple schemes (mean, median and IRWM) and leave investigation of complex combinations to future work.

\begin{figure}[!b]
{\small
\begin{center}
\begin{tabular}{|p{.95\linewidth}|}\hline
In IRWM, the final estimates from $M$ methods $e_1, e_2,...,e_m$ that have been ranked $r_1,r_2,..r_m$ is a weighted sum by the
ranks of all methods in the ensemble. The top and bottom-ranked methods of $m$ methods get a weight of $m$ and $1$ (respectively).
More generally, a method with rank $r_i$ gets a weight of $m+1-r_i$.
The final estimate in IRWM is hence $\left(\sum_i(m+1-r_i){\cdot}e_i\right)/\left(\sum_ii\right)$.\\\hline
\end{tabular}
\end{center}
}\caption{IRWM. Generalized from~\cite{Mendes2003}.}\label{fig:irwm}
\end{figure}

\begin{figure*}[!t]
\centering
{\small  
\begin{tabular}{l|l|l|l}  
\multicolumn{2}{c}{\textbf{Pre-Processors}} & \multicolumn{2}{c}{\textbf{Algorithms}} \\\hline
\textbf{Abbreviation} & \textbf{Explanation} & \textbf{Abbreviation} & \textbf{Explanation} \\\hline
norm & Normalization &  ABE0-1NN & Basic ABE with 1 nearest neighbor \\
log & Taking natural logarithm &  ABE0-5NN &  Basic ABE with 5 nearest neighbors \\
PCA & Principal Component Analysis & SWReg & Stepwise Regression \\
SFS & Sequential Forward Selection & CART (yes) & Classification and Regression Tree with pruning \\ 
SWReg & Stepwise Regression &  CART (no) & Classification and Regression Tree without pruning   \\
width3bin & Discretize into 3 bins based on equal width &  NNet & Neural Net with one hidden layer  \\
width5bin & Discretize into 5 bins based on equal width & SLReg & Simple linear regression  \\
freq3bin & Discretize into 3 bins based on equal frequency & PCR & Principal components regression  \\
freq5bin & Discretize into 5 bins based on equal width & PLSR & Partial least squares regression  \\
none & Apply no pre-processor & &  
\end{tabular}}
\caption{The summary table for the {\em solo-methods}. This table provides a list of abbreviations as well as their explanations for pre-processors and learners used in this research. 
}\label{fig:solo-methods}
\end{figure*}



\subsection{90 Solo-Methods}
In our experiments, we used 10 different pre-processors and 9 learners.
These were selected on two criteria:
\bi
\item
Learners must come from the SE effort estimation literature; e.g.  \cite{Lum2008, Mendes2003, Jorgensen2007, Shepperd1996, Kultur2008, Shepperd1997, chang74, vent93, Boehm2000}.
\item
Learners must make different 
assumptions about the data.
\ei
This second criteria is based on data-mining theory
that recommends using different learners that fail under different circumstances~\cite{Kittler1998, Alpaydin1998, Dietterich2000, Ghosh2002}.
Ensembles  work best when one member of the ensemble patches the mistakes
made by other methods of the ensemble.
We hence used 10 pre-processors:
\bi
\item Three {\em simple preprocessors}: {\bf none, norm, and log};
\item One {\em feature synthesis} method: {\bf PCA};
\item Two {\em feature selection} methods: {\bf SFS} and {\bf SWreg};
\item Four {\em discretization} methods:  Based on equal frequency/width.
\ei
and 9 learners:
%Furthermore, based on our reading of the effort estimation literature~\cite{Lum2008, Mendes2003, Jorgensen2007, Shepperd1996, Kultur2008, Shepperd1997, chang74, vent93, Boehm2000}, we identified nine commonly used learners that divide
%into
\bi
\item Two {\em iterative dichotomizers}: {\bf CART(yes), CART(no)};
\item A {\em neural net}: {\bf NNet};
\item Four {\em regression methods}: {\bf LReg, PCR, PLSR, SWReg}.
\item Two {\em instance-based} learners: {\bf ABE0-1NN, ABE0-5NN};
\ei
Note that ``ABE'' is short for analogy-based effort estimation. ABE0-kNN is a standard
analogy-based estimator with execution steps of:
\bi
\item
{\em Normalization} of data to zero-one interval;
\item
A {\em Euclidean} distance measure; 
\item
Estimates generated using the {\em k nearest neighbors}.
\ei
For detailed descriptions of all these learners, see Appendix~\ref{sect:learners-preprocessors-used-in-this-study}.

Combining $10$ pre-processors and $9$ learners results in 10*9=90 solo-methods.
A summary table for abbreviations of the methods and pre-processors is provided in \fig{solo-methods}.

New methods are being constantly invented (e.g. see~\cite{Corazza2010,Li2009,Kultur2008})
so we make no claim that the 90 methods explored here  cover the space of all possible effort estimators. 
Hence, we take care {\em not} to conclude that some particular combination of algorithms is best.
Rather our conclusions will be that, given a set of models, it is best to rank them and generate multi-model
estimators from the top-ranked models.


%The combination schemes we use in this research are mean, median and inverse ranked weighted mean (IRWM).
%The mean method takes the mean of the estimates coming from \textit{n-many} solo-methods.
%In a similar manner median method takes the median of these estimates.
%Another proposed method is the so called IRWM.
%IRWM was originally proposed as an adaptation scheme in analogy-based estimation.
%It allows higher ranked analogies to have greater influence than the lower ranked ones.
%We adapted this strategy for combining solo-methods, in the sense that the estimates of more successful solo-methods will have a greater influence.
%For example if we have $n$ estimates coming from $n$ solo-methods and these methods are ordered from the most successful to the least successful according to a performance criteria, the IRWM based multi-method's estimate is calculated as in Equation \ref{equation:irwm}, where MM stands for multi-method and SM stands for solo-method.
%
%\begin{equation}
%estimate_{MM_{i}} = \sum_{i=1}^{n}{(n+1-i)*estimate_{SM_{i}}}
%\label{equation:irwm}
%\end{equation}
%
%When combining solo-methods the number and the characteristics of methods are as important as the combination scheme~\cite{Kittler1998, Alpaydin1998}.
%To find the $n$ suitable solo-methods, we first run $90$ solo-methods on $20$ different datasets and sort them according to their percentage of losses over all 7 performance criteria (details of these criteria are in \tion{methodology}).
%Then we choose the best performing 2, 4, 8 and 16 solo-methods and combine them into multi-methods via mean, median and IRWM.
%We will treat combining top $n$ solo-methods like a pre-processor and taking mean, median or Irmw like a learner; hence, the naming convention for multi-methods become Top\textit{n}/learner (e.g. the multi method of taking IRWM of top 4 solo-methods can be named as Top4/IRWM).
%So we have \textit{4 $n$ numbers X $3$ combination methods = 12 multi-methods}.
%The list of all all solo-methods as well as all multi-methods are given in \fig{res_rank}.


%The first phase of our experimentation aims at finding out the performance of numerous {\em solo-methods} and learning their rankings.
%This ranking lets us identify and select the top performing {\em solo-methods}.
%In the second phase of our experimentation, we bring combine the selected {\em solo-methods} into a multi-learner structures (i.e. {\em multi-methods}).
%In the following paragraphs, we provide more details regarding the {\em solo-methods} as well as regarding how we form various {\em multi-methods}.


%\subsection{Pruning}
%Not all of the above combinations are valid. For example, ABE0 normalizes all numerics to the range min..max, 0..1. Hence:
%\bi
%\item
%The combination
%{\em norm-ABE0} is redundant ;
%\item
%The combination {\em none-ABE0} is misleading since ABE0 always applies normalization/
%\ei
%The following methods were therefore prunes:
%\bi
%\item XXXX
%\ei
%In all, this left YYY solo methods\footnote{Ekrem- please regenerate the charts with only these methods}.
