Where is the best effort estimator?

Jacky W. Keung

National ICT Australia Ltd. Australia CSE, University of New South Wales, Australia

Jacky.Keung@nicta.com.au Ekrem Kocaguneli, Tim Menzies

CS & EE, WVU, Morgantown, USA

ekocagun@mix.wvu.edu, tim@menzies.us

ABSTRACT

Background: Prior work on software cost estimation studies reported conflicting results as to which evaluation criteria such as MMRE should be used for effective evaluation of model prediction performance. It is becoming an eminent issue in the research of cost estimation models. Aim: Method: 9 preprocessors combined with 10 learners on 20 publicly available software project datasets to produce a massive 1,800 predictions to indicate the efficacy of different evaluation criteria. Results: Instead championing a particular evaluation criterion, our results clearly indicate the evaluation criteria being used are practically producing similar results. Conclusion: based on our finding, our recommendation is that the quality of the dataset and the preprocessor used will have an influential impact on the prediction result, different evaluation criteria examined in the study should not be used to fully appreciate the quality of the prediction model.

Categories and Subject Descriptors

Software Engineering [Software Metrics]: Data Mining

General Terms

Software Cost Estimation, Linear Regression, Regression Trees, Neural Nets, Analogy, MMRE, Evaluation Criteria

Keywords

Software Cost Estimation, MMRE, Evaluation Criteria

1. INTRODUCTION

Wilcoxon-Mann-Whitney test is a non-parametric statistical hypothesis test commonly used to compare the difference between two groups of sample data, and it is used as an alternative to the paired Student’s t-test when the population distribution cannot be assumed to be normal. It is more

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ICSE 2011 Waikiki, Honolulu, Hawaii USA Copyright 2011 ACM X-XXXXX-XX-X/XX/XX ...$10.00.

robust than the Student’s t-test as it compares the sums of ranks, unlike Student’s t-test which may introduce spurious findings as a result of presence of outliers may be existed in the given datasets.

2. RELATED WORK

[?] [?]

2.1 Mean Magnitude of Relative Error

Mean Magnitude of Relative Error (MMRE) is probably the most widely used evaluation criterion for assessing the performance of competing software prediction models. However it has flaws. Foss et al. (2003) have provided an extensive discussion using a simulation study demonstrating that MMRE does not select the best model, and they strongly recommend not using MMRE to evaluate and compare prediction models.

3. HYPOTHESIS

In this study, Our hypothesis is that

4. EXPERIMENT DESIGN

4.1 Data Pre-processors

4.2 Data Learners

5. EXPERIMENT

6. RESULT

7. DISCUSSION

8. CONCLUSION

9. REFERENCES