
\section{Learners and Pre-Processors}
\label{sect:learners-preprocessors-used-in-this-study}

\subsection{Pre-Processors}
\label{sect:pre-processors}

\begin{itemize}
\item {\bf None} is the simplest preprocessor- all values are unchanged.

\item \textbf{{\em Simple numeric techniques:}} This category lists the pre-processors that entail mere numeric alterations of actual values rather than aplication of certain algorithms.
	\begin{itemize}
		\item \textbf{norm:} \textit{``norm''} represents the application of normalization on the data. The data is normalized to 0-1 interval according to Equation \ref{equation:normalization}: 
		\item \textbf{log:} The natural logarithm of the independent variables are taken. 
		Similar to all pre-processing procedures, the dependent variable is excluded from the pre-processing method.
	\end{itemize}
	
\begin{equation}
normalizedValue = \frac{(actualValue - min(allValues))}{(max(allValues) - min(allValues))}
\label{equation:normalization}
\end{equation}

\item {\em \textbf{Feature synthesis:}} 
\begin{itemize}
\textit{``PCA''} stands for principal component analysis~\cite{Alpaydin2004}. PCA is widely used as a dimension alteration mechanism. 
The alteration in PCA occurs in the form of changing an \textit{n}-dimensional space into another \textit{n}-dimensional space.
PCA lets the user know which particular features are most influential in the new \textit{n}-dimensional space, hence user can choose to use only the most influential features.
Therefore, it is also common to see the usage of PCA like a dimensionality reduction mechanism.
However, in our study we prefer to use PCA only to change the \textit{n}-dimensional space (not the number of features).
\end{itemize}

\item {\em \textbf{Feature selection:}} Some of the pre-processors aim at finding a subset of all features according to certain criteria.
This subset is supposed to produce a feature subset that will ultimately increase the estimation accuracy. 
Under this category, we have SFS and SWREg.
\begin{itemize}
\item \textbf{SFS:} Sequential forward selection (\textbf{``SFS''}) is a feature selection mechanism, in which features are added into an initially empty set until no improvement is possible with the addition of another feature.
The so called \textit{improvement} is defined through an \textit{objective} function.
In our implementation based on MATLAB, the \textit{objective} function is the mean-squared-error of a simple linear regression on the training set.
One caution to be made here is that exhaustive search algorithms over all features can be very time consuming ($2^n$ combinations in an \textit{n}-feature dataset), therefore SFS works only in forward direction (no backtracking).

\item \textbf{SWReg:} SWReg stands for stepwise regression, which can be defined as a systematic method for adding and removing features from a multilinear model based.
Removal and addition of features depend on their statistical significance in regression. 
Our SWReg implementation that is developed by using MATLAB starts execution with a preliminary model, then it compares the performances of smaller and larger models.
At each step a potential feature is to be decided for addition or removal.
Addition and removal are based on the p-value in an F-statistic.
In a particular step, the F-statistics for two models (models with and without the feature that is being questioned in that step) are calculated.
Provided that the feature was not in the model, the null hypothesis is: ``Feature would have a zero coefficient in the model, when it is added''.
If the null hypothesis can be rejected, then the feature is added to the model.
As for the other scenario (i.e. feature is already in the model), the null hypothesis is: ``Feature has a zero coefficient''.
If we fail to reject the null hypothesis, then the term is removed. 
Basically the steps followed by the method are as follows:
\begin{itemize}
\item Fit an initial model.
\item If some features have low p-values (lower than a pre-defined threshold, which is chosen to be $0.05$ in our study), i.e. if it is likely that they would have non-zero coefficients when added to the model, then add the one with the lowest p-value, then repeat this procedure. 
If all terms have higher p-values than the threshold value, then proceed with the next step.
\item If some of the features have p-values, which are greater than an exit threshold (the exit threshold we used in our study is $0.10$ \textit{AND} $max(pValuesOfPreviousStep)$), then remove the feature with the highest p-values. 
Remember that having a high p-value means that it is likely that we will fail to reject the null hypothesis of having a zero coefficient.
\item Exit when none of the above steps improves the model.
\end{itemize}

One caution we need to make here is that, stepwise methods are locally optimal and may not necessarily be globally optimal.
In other words, depending on the initial model and the order of features for inclusion-exclusion, the algorithm may result in different final models, hence different performances.
\end{itemize}

\item {\em \textbf{Discretization:}}
\begin{itemize}
\item \textbf{width3bin:} This procedure bins each one of the data features into 3 bins, depending on equal width of all bins.
The bin-width for a general n-bin procedure is given in Equation \ref{equation:binning}.
In our 3-bin case, once we know the bin-width, we assign each feature value to either $1$, $2$ or $3$, depending on which particular bin the value is in.

\begin{equation}
\scriptsize
binWidth = ceiling(\frac{(max(allValues) - min(allValues))}{n})
\label{equation:binning}
\end{equation}

\item \textbf{width5bin:} Exactly same as \textit{``width3bin''} except that this time we have $5$ bins instead of $3$.

\item \textbf{freq3bin:} \textit{``freq3bin''} means binning each feature into 3 bins, depending on the equal frequency count (equal number of instances in each bin).
Similar to previously mentioned binning methods, in this method each feature value is assigned to $1$, $2$ or $3$.
For that, all values of a particular feature is sorted in ascending order.
Then first $ceiling(numberOf(allInstances)/3)$ instances are assigned $1$, the second $ceiling(numberOf(allInstances)/3)$ instances are assigned $2$ and so on.

\item \textbf{freq5bin:} Exactly same as \textit{``freq3bin''}, only this time we have $5$ bins.
\end{itemize}

\item {\em \textbf{Others:}} The option(s) that cannot be categorized into afore mentioned categories are mentioned here.
\begin{itemize}
\item \textbf{none:} Application of no pre-processing to the data is also a choice. \textbf{``none''} represents the choice of no pre-processor selection.
\end{itemize}

\end{itemize}

\subsection{Learners}
\label{sect:learners}



\begin{itemize}
\item {\em \textbf{Instance-based learners:}}  When we refer to instance-based learners (either ABE methods or nearest-neighbor based methods), there are various design options to be decided and if not clearly stated those decisions remain vague to the reader.
The analogy based estimation method that we call ABE0-\textit{x}NN refers to a very basic type of ABE that we defined through our readings of various ABE studies~\cite{Mendes2003, Li2009, Kadoda2000}.
In ABE0-\textit{x}NN, features are firstly normalized to 0-1 interval, then the distance between test and train instances is measured according to Euclidean distance function, \textit{x} nearest neighbors are chosen from training set and finally for finding estimated value (a.k.a adaptation procedure) the median of \textit{x} nearest neighbors is calculated.
Therefore, when we say ABE0-xNN, all the design options including the choice of \textit{x-many} closest analogies become explicit to a reader. 
In our experimentation we use two different \textit{x} values (i.e. two different analogy values):
\begin{itemize}
\item \textbf{ABE0-1NN:} Only the closest analogy is used. 
Since the median of a single value is itself, the estimated value in ABE0-1NN becomes the actual effort value of the closest analogy in the training set.
\item \textbf{ABE0-5NN:} Five closest analogies are found and used for adaptation.
\end{itemize}

\item {\em \textbf{Iterative dicotomization:}} Under this category, we use classification and regression trees (\textit{CART}) as described by Breiman et al.~\cite{Breiman1984} and developed it using MATLAB routines.
CART is a non-parametric technique and it can work both for classification and regression type of problems, in our case it is the latter.
There are in fact a variety of approaches for designing CART. 
When planning to construct a CART, there are a couple of points to consider:

\begin{itemize}
\item Selection of splits
\item Decision on when a node is pure enough (when to stop splitting any further, i.e. finding the terminal node)
\item Assigning a class to each terminal node
\end{itemize}

Firstly, the selection of splits: There are again a variety of solutions that are used to decide on the selection of splits such as misclassification rate, information (or entropy) or GINI index. 
CART uses the GINI index to calculate the impurity of a tree~\cite{Breiman1984}.
Secondly, decision on when to stop further splits: The implementation allows you to specify a threshold value or use the default value.
We have used default threshold values in the implementation (for further details please refer to MATLAB documentation).
Lastly, assigning class to each terminal node: Since our problem in effort estimation is a regression problem, we have no classes but actual numeric values in the terminal nodes. Each
test instance results in a terminal node and therefore different test instances resulting in the same terminal node are given the same predicted value. For regression, the predicted value in a terminal node is a fit on the independent variables of the instances in that node. 

We have used 2 types of CART, which are given below:
\begin{itemize}
\item \textbf{CART-On:} In this version, the pruning is on, meaning that the training data is used in a cross validation process and for each cross validation run, some internal nodes are chosen as the leaf nodes and their sub-nodes are removed.
Finally, the sub-tree that resulted in the lowest error rate is returned. 
The returned sub-tree is a sub-optimal solution and it is locally optimum.
\item \textbf{CART-Off:} The sub-trees of CART will not be considered and the full tree will be used.
\end{itemize}

\item {\em \textbf{One-layered neural net:}} \textit{``Neural Nets''} (from now on NNet) are memoryless structures that can be defined as universal approximators~\cite{Hornik1989}, meaning that: 1) They store information coming from training data as weights during training and after that phase they do not need training data and 2) they can approximate any function depending on how complex the NNet structure is.
NNets basically have a 3 layer structure: Input layer, hidden layer and the output layer.
Depending on the complexity of the problem, one can model more complex functions with a NNet by increasing the number of hidden layers in the structure.
In our implementation based on MATLAB, we used a simple NNet structure with 1 hidded layer, 1 input layer and 1 output layer.

\item {\em \textbf{Regression methods:}} Under this category, we list our regression-based learners.
\begin{itemize}
\item \textbf{LReg:} LReg stands for linear regression. 
Given the dependent variables, this learner calculates the coefficient estimates of the independent variables.
\item \textbf{PCR and PLSR:} Partial Least Squares Regression (PLSR) as well as Principal Components Regression (PCR) are algorithms that are used to model a dependent variable and we have used MATLAB routines to implement those functions in our experiments.
While modeling an independent variable, they both construct new independent variables as linear combinations of original independent variables.
However, the ways they construct the new indepent variables are different:
\begin{itemize}
 \item \textbf{PCR:} This method generates new independent variables to explain the observed variability in the actual ones.
However, while  generating new variables the dependent variable is not considered at all.
In that respect, PCR is similar to selection of \textit{n-many} components via PCA (the default value of components to select is 2, so we used it that way) and applying linear regression.
\item \textbf{PLSR:}  Unlike PCR, PLSR considers the independent variable and picks up the \textit{n-many} of the new components (again with a default value of 2) that yield lowest error rate.
Due to this particular property of PLSR, it usually results in a better fitting.
However, the question of which method is more parsimonious still remains context dependent.
\end{itemize}
\item \textbf{SWReg:} We have previously defined SWReg and mentioned that it was a special regression algorithm.
We have also stated how it can be used as a feature selection algorithm.
In the algorithms section, we use SWReg as a regression method on the \textit{selected-out} independent variables to explain the dependent variable.
\end{itemize}

\end{itemize}
