\section{Threats to Validity}
\label{sect:threats_to_validity}

{\em External validity} questions whether the results can be generalized outside
the specifications of a study\cite{Milic2004}.  
To ensure external validity, this paper has studied a large number of projects.
%Our data sets are diverse, measured in terms of their
%sources, their domains and the time they were developed in.
%We use datasets composed of software development
%projects from different organizations around the world to
%generalize our results\cite{Bakir2009}. Our reading of the literature
%is that this study uses more data, from more sources,
%than numerous other papers. 
For example,
Table 4 of~\cite{Mendes2007} lists the total number of projects used by a sample of
other studies. The median value of that sample is 186; i.e.  one-sixth
of the 
1198 projects used here.
In terms of external validity, this report has higher validity than a standard effort
estimation study (which might apply  one error measure to a handful of data sets).

It is clear that this study has not explored a full range of effort estimation algorithms.
Future work is required  to repeat this study using the top performing solo-methods found here and possibly more.
Nevertheless, given the extent of the experimentation reported here,
this paper offers much support for the general claim that 
when generating estimates,
it is wise to {\em first} sort and prune all available solo-methods, and {\em then} try combinations of the best solo-methods.

{\em Construct validity} (i.e. face validity) asks if we are measuring
what we actually intended to measure~\cite{Robson2002}.
Previous studies have concerned themselves with the construct validity of 
different performance measures for effort estimation (e.g. \cite{Foss}).
To make sure that we do not favor a particular conclusion due to limited number of performance measures, we used $7$ error measures and questioned the stability of our conclusions.
We showed empirically the surprising result that multi-methods are stable across a range of performance criteria.

In terms of {\em internal validity}, there is one 
dimension of experimental conditions not explored above.
This paper used leave-one-out to assess methods. An alternate 
experimental condition would be to use 
N-way cross-validation 
where examples are divided into $B$ bins and each  bin is tested on a model learned from
the remaining bins (typical values for $B$ are 3 or 10).
In theory, not using cross-validation is a threat to the validity of our results since we did not check if
our results were stable across both leave-one-out and cross-validation.
%According to standard results in the literature~\cite{breiman92,Kohavi95,Hastie08}, they have different
%bias and variance properties:
%\bi
%\item {\em Bias} is the distance of mean estimate from the actual value (so low biased estimators fall close to actuals);
%\item {\em Variance} is the size of the space of estimates around the mean (so low variance estimators generate predictions closer to the mean).
%\ei
According to Hastie et al.\cite[p242]{Hastie08},
leave-one-out testing
generates:
\bi
\item
Lower bias estimates than cross-validation (since cross-validation must learn from fewer examples);
\item
Higher variance estimates than cross-validation (since leave-one-out conducts many more tests).
\ei
Similar assertions are made by Kohavi~\cite{Kohavi95} and Breiman~\cite{breiman92}.

Elsewhere~\cite{ekrem11}, we have documented the surprising result that,
at least for the data sets of \fig{datasets},
there is very little difference in the bias and variance values generated for leave-one-out and
cross-validation. 
%For example, \fig{bias_variance} 
%\begin{figure}[!t]
%\begin{center} \includegraphics[width=3in]{plots/biasVarNasa93.pdf} \end{center}
%\caption{90 logged bias/variance results for leave-one-out (LOO) and 3-Way cross-validation for Nasa93. }
%\label{fig:bias_variance}
%\end{figure}
%shows the bias/variance results seen for the 90 solo-methods in NASA93. Note that, contrary
%to the discussion above, leave-one-out does not have a lower bias and higher variance than cross-validation
%(statistically, the two distributions are indistinguishable via a 95\% Wilcoxon test).
%Similar results were seen for all the other data sets of \fig{datasets}.
%
%Why are the effort data sets performing differently to the predictions of Kohavi, Breiman, Hastie et al.?
%Our answer to this question is complex and is
%the subject of another paper~\cite{ekrem11}. For now, we offer one comment.
%With the exception of the CHINA data set,
%Kohavi \& Breiman \& Hastie et al. explore data sets that are far larger than those of \fig{datasets}.
We conjecture that the standard comments on ``bias vs variance trade-off'' for ``leave-one-out vs cross-validation''
needs to be revisited for the small effort data sets seen in effort estimation.

Since leave-one-out and cross-validation have similar bias/variance profiles, we are free to select either method.
This paper uses leave-one-out for the following reasons.
Leave-one-out is a deterministic procedure that can be exactly repeated by any other researcher
with access to a particular data set.  The same cannot be said about  cross-validation since it
 divides the data into bins using:
\bi
\item
A random number
generator;
\item
Some stratification heuristics.
Stratification attempts to maintain the same class distributions in each bin as in the entire data set). Stratification
is a heuristic procedure since it may encounter issues that have no singular solution (e.g. how to space
rare outliers amongst the bins).
\ei
Note that without knowledge of the stratification heuristics, and without access
to an identical random number generator, it can be difficult for researcher ``A'' to reproduce the cross-validation
results of researcher ``B''.


