\section{Conclusion}
There are many effort estimation methods and little guidance on how to choose between them.
Prior results from Shepperd et al. are pessimistic about being able to rank different methods. If $M$ methods are ranked in an experiment,
then that ranking can be changed by altering the conditions of that experiment (e.g. the data used in training or the performance measure used).

This report confirms the ranking instability effect reported by Shepperd et al.: in \fig{rank_changes} and \fig{rank_changes_moe} it was seen
that a method's ranking can change by some amount ${\delta}r$. 
Nevertheless, our conclusions are more optimistic than those of
Shepperd et al. While some methods have very large ${\delta}r$, others do not.
In fact, for the solo-methods shown in \fig{rank_changes}, the better methods have smaller ${\delta}r$. Better yet, when we combined
the highest ranked solo-methods, we found that the top-ranked multi-method had almost zero ${\delta}r$. Therefore, we recommend multi-methods
since, as shown in \fig{rank_changes_moe}:
\bi
\item
The multi-methods consistently out-perform most, if not all, of the solo-methods;
\item
The performance of the multi-methods that use the larger ensembles is more trustworthy (has smallest ranking instability).
\item
Also, as shown in \fig{mdmre-spectrum}, the multi-methods avoid the problem of very
large errors seen with other methods.
\ei 
%A general overview of the results was provided in the previous section.
%Now we are going to investigate them deeper and map them to particular research questions that guided this research.
The success of our results raises the question: Why were prior experiments in ensembles for effort estimation~\cite{Baker2007,Kocaguneli2009} so unsuccessful?
Reading from prior effort estimation literature, we offer the following hypothesis: {\em Many effort estimation methods are inferior} so combinations of
inferior members will also  be inferior.
In \fig{rank_changes}, for example, while there is evidence for 13 useful methods (in the range $1 \le X \le 13$), the remaining methods have
large ${\delta}r$ or fall into the worse ranks. That is, of the 90 solo-methods we have explored, there is no evidence for the value of
$\frac{77}{90}=86\%$ of the methods. Without a ranking like \fig{rank_changes} to guide method selection, it is therefore probably that researchers
will try combining weaker methods.

Finally, we can offer an answer to the vexing question: \textit{What is the best effort estimator?}
Simply put, we have no evidence suggesting that any  solo-method is some universal, best in all circumstances, effort estimator.
However, if no solo-method is always best, some ensemble of solo-methods may offer consistently better performance.
We have seen that 9 out of 12 multi-methods outperformed all others. Better yet, our top-ranked multi-method has the greatest
stability among any of the 102 methods explored in this study.
Hence, this study proposes multi-methods as the technique that finds the  best learner.

