Estimating the effort required for software projects is a difficult task in software project planning.  There are multiple algorithms to estimate effort from previous projects, but not a standard criteria upon which to evaluate the algorithms accuracy.  This paper tests multiple effort estimation algorithms using multiple error measures from the literature, to see if different error measures provide a stable ranking of algorithms.  The results indicate instability in rankings between different error method, suggesting a need for more robust ways of algorithm comparison for effort estimation.