Noname manuscript No. (will be inserted by the editor)

Kernel Methods for Software Effort Estimation

Effects of different kernel functions and bandwidths on estimation accuracy

Received: date / Accepted: date

Abstract Analogy based estimation (ABE) generates an effort estimate for a new software project through adaptation of similar past projects (a.k.a analogies). Majority of ABE methods follow uniform weighting in adaptation procedure. In this research we investigated non-uniform weighting through kernel density estimation. After an extensive experimentation of 5 kernels, 5 bandwidth values and 330 ABE variants, we found 2 important results: 1) Right number of analogies is vital in ABE and 2) non-uniform weighting through kernel methods never outperformed uniform weighting. Hence, -provided that similar experimental settings are adopted- we discourage the use of kernel methods as a weighting strategy in ABE.

Keywords Effort estimation, data mining, kernel function, bandwidth

1 Introduction

Software effort estimates are reported to be often wrong by a factor of four [?] or even more [?]. Wrong estimates can have critical results for a company such as cancellation of budget over-running projects, thereby wasting the entire effort. Therefore, effort estimation is an active research area [?, ?, ?, ?, ?] that constantly explores more variations.

ABE is an active area of research in effort estimation and is based on the premise that effort of a future project can be estimated by adapting the effort values of k similar past projects ( analogies) [?, ?, ?]. Among proposed adaptation methods we can name choosing closest analogy [?, ?], taking mean or median of k analogies [?, ?]. Taking mean and median are examples of uniform weighting ABE methods (from now on U-ABE), i.e. the low ranked analogies have just as much influence as the high ranked analogies. Another method to let different analogies have different weights. For example Mendes et. al. proposes a method called inverse rank weighted mean (IRWM) that allows higher ranked analogies to have greater influence [?, ?]. Such methods are examples of non-uniform weighting ABE methods (from now on N-ABE).

In this research, we contribute narrowing down the search space in software effort estimation by investigating different weighting strategies in ABE. We investigate the impact of non-uniform weighting and use kernel density estimation [?] that is reported to be one of the most popular non-parametric estimators that can uncover structural properties in the data [?, ?, ?, ?].

Address(es) of author(s) should be given

To guide us in this paper, we have identified the following research questions: RQ1 Is there any evidence that non-uniform weighting improves the performance of

ABE? RQ2 What is the effect of different kernels for non-uniform weighting ABE? RQ3 What is the effect of different bandwidths when used for non-uniform weighting

ABE? RQ4 How do the characteristics of software effort datasets influence the performance of

kernel weighting in non-uniform weighting ABE? RQ5 Which parameters among kernel, bandwidth and selected analogy number matter

the most in terms of estimation accuracy?

After an extensive study of 330 ABE variants, it is clear that selecting the right number of neighbors is vital for effort estimation. On the other hand, N-ABE methods are never enhanced by a particular kernel method. These negative results have at least three positive consequences:

1. There is nothing inherently wrong with intuition-based weighting schemes like IRWM (since all the weighting schemes we explored had similar results).

2. We can better focus future research: Our reading of the results is that the value of k appears to be a more important factor in ABE than non-uniform weighting strategies.

3. Unlike studies in other domains that claim “kernel does not matter but the bandwidth does”, we cannot offer supportive evidence for a statistical heuristic that leads to the same conclusion. Our results point to the conclusion that neither kernel nor the bandwidth has a considerable impact for improvement in software effort estimation

Therefore, we do not recommend weighting via kernel methods to as a productive area of research to explore in ABE. We therefore advise researchers in this area to explore other aspects of ABE. Note that in accordance with the invaluable feedbacks of the reviewers, the paper is intended to be a conscise report of our results rather than an in depth analysis. Therefore, reader is suggested to contact the authors or follow the references for in-detail explanation of certain concepts.

The rest of the paper is organized as follows: In Section 2 we provide background information regarding related work. Section 3 explains the adopted experimental methodology. In Section 5 we provide a brief discussion of our pre-experimental intuition regarding N-ABE. In Section 4 results are presented and in Section 6 the threats to the validity of these results are discussed. In Section 7 we summarize our conclusions and answer the research questions. Finally in Section 8 we list future directions of this research.

2 Background

2.1 Software Effort Estimation

We can divide software effort estimation into at least two groups [?]: expert judgment and model-based techniques. Expert judgment methods depend on consolidation of expert opinions regarding the cost of a new project and are widely used in software effort estimation practices [?]. Expert judgment can be applied either explicitly (following a method like Delphi [?]) or implicitly (informal meetings among experts). Unlike expertbased methods, model-based techniques do not rely heavily on human judgment. Model based techniques are products of:

1) Algorithmic and parametric approaches or

2) Induced prediction systems.

The first approach is the adaptation of an expert-proposed model to local data. A widely known example to such an approach is Boehm’s COCOMO method [?]. The second approach is particularly useful in the case where local data does not conform to the specifications of the expert’s method. A few examples of induced prediction systems are linear regression, neural nets, model trees and analogy based estimation [?, ?].

Analogy based estimation (ABE) or estimation by analogy (EBA) is a form of case based reasoning (CBR) and it is grouped together with induced prediction systems. ABE generates its estimate for a new project by gathering evidence from similar past projects. When we analyze the previous research of experts on the domain of ABE such as Shepperd et. al. [?], Mendes et. al. [?] and Li et. al. [?], we can see a baseline technique lying under all ABE methodologies. The baseline technique is composed of the following steps:

– Form a table (training set) whose rows are completed past projects and whose columns are independent variables (the features that define projects) and a dependent variable (the recorded effort value).

– Decide on the number of similar projects (analogies) to use from the training set, i.e k-value.

– For each test instance, select k analogies out of the training set.

– While selecting analogies, use a similarity measure (for example the Euclidean distance).

– Before calculating similarity, apply a scaling measure on independent features to equalize their influence on this similarity measure.

– Use a feature weighting scheme to reduce the effect of less informative features.

– Adapt the effort values of the k nearest analogies to come up with an effort estimate.

Following the steps of this baseline technique, we define a framework called ABE0. ABE0 uses the Euclidean distance as a similarity measure, whose formula is given in Equation 1. In Equation 1, w_{i} corresponds to feature weights applied to independent features. ABE0 framework does not favor any features over the others, therefore each feature has equal importance in ABE0, i.e. w_{i} = 1.

Distance = \sqrt{∑_{i=1}^{n} w_{i}(x_{i} − y_{i})^{2}}  (1)

The next step is deciding on how to adapt project costs. There is a wide variety of adaptation strategies in the literature [?]. Using effort value of the nearest neighbor [?], taking mean [?] or median [?] of closest analogies, inverse distance and inverse rank weighted mean of closest analogies are among the commonly used adaptation methods [?]. Angelis et. al. suggests that as the number of the closest projects increase, median is a robust solution [?]. They have found that taking median instead of mean decreases the estimation error. We want the estimates of ABE0 framework to represent the majority of selected instances and not greatly affected by extreme values. Therefore, ABE0 returns the median effort values of the k nearest analogies. Since ABE0 implicitly assigns equal weights to k nearest analogies, it turns out to be an U-ABE method.

In this research we will compare the results of ABE0 framework with different non-uniform weighting strategies, i.e. with different N-ABE methods. N-ABE methods have been previously adressed in literature. For example inverse rank weighted mean (IRWM) was proposed by Mendes et. al. [?]. IRWM method enables higher ranked analogies to have greater influence than the lower ones. Assuming that we have 3 analogies, the closest analogy (CA) gets a weight of 3, the second closest (SC) gets a weight of 2 and the weight assigned to the last analogy (LA) is 1.

2.2 Kernel Density Estimation

IRWM is one example of a broad class of statistical reasoning called kernel density estimation, where IRWM acts like a triangular kernel assigning weights to analogies on the basis of their distance. However, IRWM-like methods have their roots in expert judgment. In other words, in the lack of valuable experts, such a weighting strategy would be almost impossible to apply to the needs of a particular dataset. Kernel density estimation on the other hand is a non-parametric estimation method that is used to uncover the underlying structures of data, which a parametric approach may fail to reveal [?] and it requires no expert intervention. Since we used the univariate kernel density estimation, we will suffice to mention the univariate case in this paper. However, the same approach can be easily adapted to higher dimensionalities [?, ?].

The kernel function is usually chosen to be unimodal and symmetric about zero [?], such as a probability distritibution function. In a kernel estimation method, the center of the kernel is placed right on each data point and the influence of data points is distributed to the overall neighborhood. To reach the final density function, contributions coming from all data points are summed up. For example, note how individual Gaussian curves add up to generate the final density estimate in Figure 1 (one kernel is added for each observation along the x-axis [?]).

Kernel density estimation has been successfully used for different type of datasets. For instance Palpanas et. al. use kernel density estimation to address the problem of deviation detection in environment of sensor networks [?]. Frank et. al. use kernel estimation for locally weighting the attributes of Naive Bayes [?]. Furthermore John et. al. use kernel estimation to tackle the normality assumption regarding continuous datasets [?]. They replace single Gaussian distribution that is used to model continuous data with non-parametric kernel density estimation and they report considerable improvements on real and artificial datasets. Although kernel density estimation is used in different areas for modeling different types of data, to the best of our knowledge it was not previously used in the context of ABE. In this research we propose using kernel density estimation as a N-ABE method for assigning weights to selected analogies.

The kernels we use in our research are: Uniform, triangular, Epanechnikov and Gaussian. We can use a generic formula for some kernels, which is given in Equation 2, where 1_{(|x|<1)} is the indicator function. Furthermore, Equation 3 and Equation 4 explain for the calculation of other functions in Equation 2. Depending on the value of p in Equation 2, we can derive different kernels. For example for p = 0 we elicit the uniform kernel, for p = 1 we elicit Epanechnikov kernel etc.

Fig. 1: We see a Gaussian kernel density estimate built on individual data points. Each point is in the center of a kernel and its effect is distributed to its neighborhood. The sum of all kernels make up the final Gaussian kernel density estimate.

K(x, p) = \frac{(1 − x^{2})^{p}}{2^{2p+1}B (p + 1, p + 1)} 1_{(|x|<1)}  (2)

B (p + 1, p + 1) = \frac{Γ (a) Γ (b)}{Γ (a + b)}  (3)

Γ (n) = (n − 1)!  (4)

This paper explores the kernels of Figure 2 as well as IRWM [?, ?]. The general shapes of these kernels are given in Figure 3 [?]. IRWM is not actually proposed as a kernel method and it does not fully conform to the definition of standard kernel methods (being unimodal and symmetric about zero and so on). However, due to the weighting strategy it proposes we can read it as an expert proposed kernel, whose shape would look like the right part of a triangular kernel as in Figure 3(e).

Initially, we planned to explore many more kernels than those listed above. However, to our surprise, we found that estimation accuration was not influenced by the kernel method. A literature review revealed that a similar effect has been reported in domains other than effort estimation. A standard result is that the selection of bandwidth (h) for kernels is more influential than the kernel types [?,?]. Bandwidth h is fundamentally a scaling factor that controls how wide probability density function will spread around a point and the choice of h determines how smooth or rough density estimation will be, i.e. appropriate choice of h is critical to avoid under and over-smoothing [?, ?]. To avoid both under and over-smoothing conditions we used various bandwidth values in our research. One of the bandwidths we used is suggested by John et. al., which is h = 1/\sqrt{n} where h is the bandwidth and n is the size of dataset [?]. The other bandwidth values we used are: 2, 4, 8 and 16.

We can see how choosing different bandwidth values affect kernel density estimation in Figure 4. Figure 4 from [?] is suggestive, but not convincing evidence that effort estimation may not be improved by the selection of the kernel method. Prior work (reporting that bandwidth h values were more important than choice of kernel) comes from (e.g.) the signal processing literature [?, ?]. Data sets in those domains are very different to the data sets seen in effort estimation: thousands to billions of instances (for signal processing) versus dozens to hundreds of instances (for effort estimation). In data-started domains like effort estimation, it may be the case that an intelligent selection of the kernel could compensate for data scarcity. Note that other authors in the field of effort estimation have also shared this opinion [?, ?]. However, we are unaware of any effort estimation publications that explore this issue in a systematic manner. Hence, the rest of this paper.

Kernel Type  Formula
Uniform Kernel  K(ρ) = \frac{1}{2} 1_{(|ρ|<1)}
Triangular Kernel  K(ρ) = (1 − |ρ|) 1_{(|ρ|<1)}
Epanechnikov Kernel K(ρ) = \frac{3}{4} (1 − ρ^{2}) 1_{(|ρ|<1)}
Gaussian Kernel  K(ρ) = \frac{1}{\sqrt{2π}} e^{\frac{−1}{2} ρ^{2})}

Fig. 2: The formulas for different kernels used in this study. In formulas ρ = \frac{x−X_{i}}{h}. Note that IRWM kernel has different characteristics and its calculation details were provided in Section ??.

3 Methodology

In this section we provide the methodology that we adopted in our research. We discuss how we use kernel density estimation as a non-uniform weighting method as well as which kernels we use for this purpose. Furthermore, we provide information regarding the datasets we used in this research and discuss their characteristics. Also we provide information regarding the experimental settings we adopted. Finally we discuss the performance criteria according to which we compare the performance of N-ABE methods to basic U-ABE method ABE0.

3.1 Weighting Method

Here we summarize how kernel density estimation is employed as a non-uniform weighting method in this research. Assume that our dataset is divided into two sets: A = {x_{1}, ..., x_{k}} (selected Anologies) and R = {t_{1}, ..., t_{n−k}} (Rest of the dataset). We build the kernel density estimation on R and evaluate the resulting function at instances of A. Equation 5 shows the probability calculation with kernel density estimation. In Equation 5 the kernel K is built on training data t_{i} ∈ R and is evaluated at k^{th} analogy x_{k} for a bandwidth of h.

f(x_{k}, h) = \frac{1}{nh} ∑_{t_{i}∈R} K (\frac{x_{k} − t_{i}}{h} )  (5)

The general idea of this approach is that selected k analogies for a test instance come from a distribution and this distribution is specific to the dataset. Furthermore, according to this specific distribution we get different probability values for each analogy. In other words, we have different f(x_{k}, h) values for each analogy. We use these probability values as weights for analogies. Note that before using a probability value as a weight, we need to scale it to 0-1 interval according to Equation 6, where x_{k} represents all the analogies in A except x_{i}.

weight_{x_{i}} = \frac{f(x_{i}, h) − max(f(x_{k}, h))}{max(f(x_{k}, h)) − min(f(x_{k}, h))}  (6)

After calculating weight_{x_{i}} for each analogy, we update their actual effort values according to their weights. Updating the actual effort values simply means multiplying an effort value with its weight. Equation 7 shows how actual effort value of an analogy, x_{i}, is updated.

updatedEffort_{x_{i}} = actualEffort_{x_{i}} ∗ weight_{x_{i}}  (7)

3.1.1 Uniform vs. Non-Uniform Weighting

In this research we compare two different weighting strategies in ABE: Uniform weighting and non-uniform weighting. For the ease of naming we use abbreviations to refer to these strategies. Uniform weighting ABE is abbreviated by U-ABE, whereas Nonunform weighting ABE is abbreviated by N-ABE.

For both strategies we can come up with a number of different methods. As for U-ABE, we defined a base method that we call ABE0 (see Section ??). ABE0 assumes that each one of the k analogies have equal importance: It assigns equal weights to all analogies. Therefore, ABE0 uses the actual effort values without applying any change during adaptation for the final estimate. In terms of N-ABE methods, we use different kernel types in the context of kernel density estimation. Depending on the underlying characteristics of a particular dataset, kernel density estimation assigns certain probability values to analogies. These probability values define the importance of each analogy: Higher probability means higher weight. Different kernel types used as N-ABE methods in our research are: Uniform kernel, triangular kernel, Epanechnikov kernel, Gaussian kernel and IRWM.

The fundamental difference between N-ABE methods and ABE0 is that in ABE0 analogies are given uniform weights and their actual effort values are used in an as is manner, whereas in N-ABE methods analogies are assigned different weights and their actual effort values are multiplied by these weight values.

One point that needs further clarification is the use of uniform kernel as a N-ABE method. Since uniform kernel also follows a uniform weighting scheme, it may be confused with ABE0. However, the uniform weighting scheme of ABE0 and that of uniform kernel are quite different from one another. In uniform kernel, depending on how we select bandwidth some of the instances get uniform and non-zero probability values, whereas some others get a probability value of zero. Therefore, uniform kernel in fact turns out to be a N-ABE method.

Figure 5 succinctly illustrates the difference between uniform kernel being a N-ABE method and ABE0 being a U-ABE method. In Figure 5 x-axis shows the instance IDs, whereas y-axis shows probabilities of instances simulated by a normal probability distribution function (PDF). As we can see from Figure 5, ABE0 would assume equal importance of all instances and hence assign equal probabilities. On the other hand, a uniform kernel assigns equal probabilities to only a certain portion of the instances.

3.2 Data

In our research, we have used three commonly used datasets in software effort estimation research: Nasa93, the original Cocomo81 [?], and Desharnais [?]. Cocomo81 and Nasa93 datasets contain projects developed in NASA, whereas Desharnais dataset contains projects developed by Canadian software houses. A number of studies using these datasets subject them to pre-processing steps such as feature and instance selection. Since observing the impact of instance and/or feature selection is not one of our aims, in our research we did not resort to these pre-processing steps and used all features as well as all the instances in the datasets. Furthermore, studies making use of Desharnais dataset exclude 4 of the project instances due to the fact that some of the project properties are missing. Yet, the datasets are already sparsely populated and there are ways to handle missing entries. Simple mean imputation is one of these methods, in which missing properties are replaced with the mean values of other projects for the same property [?]. To handle instances with missing entries in Desharnais dataset, we used simple mean imputation.

Apart from selecting commonly used datasets, we took the quality of the datasets into consideration. In order to evaluate the goodness of datasets, Kitchenham and Mendes propose a quality scoring that consists of four values: poor (less than ten projects), fair (between ten to twenty projects), good (between twenty to forty projects) and excellent (more than forty projects) [?]. Following this quality criteria all the datasets we use in our research rank as excellent quality. The details regarding these datasets can be found in Figure 6.

3.3 Experiments

Our experimental settings aim at comparing the performance of standart U-ABE (ABE0) to that of N-ABE (non-uniform weighting through different kernel types). To separate train and test sets we use leave-one-out method, which entails selecting 1 instance out of a dataset of size n as the test set and using the remaining n−1 instances as the training set. For each test instance, we run ABE0 and store the estimated effort for that test instance. Then we run different N-ABE methods for the same test instance and store their estimates as well. As the analogy number is reported to play a critical role in estimation accuracy [?], both for U-ABE and N-ABE methods, we tried different numbers of analogies. Furthermore, to hinder any particular bias that would come from the settings of a single experiment, we repeated the afore mentioned procedure 20 times.

In this research we use 2 forms of ABE methods (uniform and non-uniform weighting) induced on 3 datasets (Cocomo81, Nasa93 and Desharnais) for 5 different numbers of analogies, a.k.a k values. The k values we used in our research are: k ∈ {1, 3, 5, 7, 9, dynamicK}. dynamicK is selected for each individual test instance through a process, in which we randomly pick up 10 instances from the training set as the validation set and select the lowest error yielding k value as the dynamicK. Furthermore, we use 5 different kernels (Uniform, triangular, Epanechnikov, Gaussian and IRWM) with 5 bandwidth values in N-ABE experiments. Therefore, to further explore field of software effort estimation, we investigate a total of 330 different settings in this research:

– U-ABE Experiments: 15 settings – 3 datasets * 5 k values = 15

– N-ABE Experiments: 315 settings – Standard Kernels: 3 datasets * 5 k values * 4 kernels * 5 bandwidths = 300 – IRWM: 3 datasets * 5 k values = 15

3.4 Performance Criteria

To observe the effect of non-uniform weighting in ABE, we use the following performance measures: the magnitude of relative error (MRE), median magnitude of relative error (MdMRE), prediction within 25% error rate (Pred(25)) and win-tie-loss values generated by a statistical test (Mann-Whitney U Test). MRE is used by the authors because it is the most commonly used performance criterion for assessing the performance of competing software effort estimation methods [?,?,?]. Furthermore, as we can see from Formula 8, MRE value is a direct measure of the absolute difference between the prediction and actual value [?] and hence it gives a per-instance based performance evaluation.

MRE = \frac{|actual_{i} − predicted_{i}|}{actual_{i}}  (8)

MdMRE has emerged as one of the de facto standard evaluation criterion for cost estimation models [?]. Often software effort studies make use of mean magnitude of relative error (MMRE) as the performance criterion as well. MMRE is the mean of all MRE values. However, the mean approach considers every observation and is sensitive to individual predictions that have high MREs [?]. One way to address this problem is the median approach via MdMRE. Median also gives information about central tendency, but it is less sensitive to extreme MRE values. Therefore, while we comment on the results of MRE-based measures in Section 4.1, we provide MdMRE values. The formulas of MdMRE is given in Equation 9, where n is the test set size.

MdMRE = median(MRE_{1}, MRE_{2}, ..., MRE_{n})  (9)

Pred(l) is also a very widely used performance criterion in software effort estimation studies [?]. Pred(l) is defined as the prediction at level l, which measures the percentage of estimates that are within l percent of the actual effort values [?]. The suggestion regarding the value of l is that it shall be set at 25 percent [?] and that a good prediction system should offer this accuracy level 75 percent of the time [?,?]. Therefore, we will also use Pred(l) at 25 25% (i.e. Pred(25)).

Note that, MRE related measures are subject to many pitfalls. If MRE is used a stand-alone performance evaluation criterion (i.e. not combined with appropriate statistical tests), it may lead to biased or even false conclusions [?]. To prevent us from falling into MRE-related pitfalls, we use another performance criterion called win-tie-loss calculation. A win-tie-loss calculation tells that comparison between two methods i and j makes sense only if they are statistically different. If there is no statistically significant difference between two methods, say method i and method j, then it indicates that results are observations coming from the same distribution. Therefore, methods are said to ‘‘tie” and their tie values (tie_{i} and tie_{j}) are incremented. However, if there is a statistical difference between two methods, then the method with a lower median MRE score, say i, is said to have a “win” and the one with the lower MRE, say j, is said to have a “lose”. The related values win_{i} and loss_{j} are incremented by one. The pseudocode for a win-tie-loss calculation is given in Figure 7. For the comparison of methods in win-tie-loss calculation, a non-parametric statistical test (the Mann-Whitney rank-sum test) is used at a significance level of 95%.

4 Results

As we mentioned earlier, we will evaluate the effect of non-uniform weighting against uniform weighting in an ABE context according to three performance measures: MdMRE, Pred(25) and Win-tie-loss values. In this section we present our results. We first evaluate the MdMRE and Pred(25) results for each dataset and then present the win-tie-loss values.

4.1 Evaluation of MdMRE and Pred(25) Results

The MdMRE and Pred(25) values of kernel weighted N-ABE for Cocomo81, Nasa93 and Desharnais datasets are provided in Figure 8, Figure 9 and Figure 10 respectively. In these figures N-ABE settings are indicated with a +N sign, where +N stands for N on-uniform weighting. Furthermore, different number of analogies used in each treatment are represented by actual k values, whereas dynamic k is represented by a d symbol.

4.1.1 Results for Cocomo81

In Figure 8 we see the MdMRE and Pred(25) values for Cocomo81 subject to different treatments. The first observation we can make out of Figure 8 is that for the same k value N-ABE methods fail to improve ABE0. For all settings, uniform weighting yields lower error rates. Furthermore application of different kernels for non-uniform weighting does not produce a recognizable pattern in terms of MdMRE and Pred(25). Change of bandwidths for kernels does not create a recognizable pattern in the results either. Also another observation we can make out of Figure 8 is that both for uniform and non-uniform weighting, smaller k values yield lower MdMRE and Pred(25) values. Regarding Figure 8, note that we use all 63 instances of Cocomo81 dataset with full range of its features in all treatments.

4.1.2 Results for Nasa93

Figure 9 lists the MdMRE as well as Pred(25) results for Nasa93 dataset. As we can see from Figure 9, different kernel types generate very similar results for various k values. In other words, change of kernel does not have a considerable effect on the performance of N-ABE. Furthermore, small performance changes due to different kernels do not follow a particular pattern.

Like the change of kernel, changing bandwidth has almost a non-existent effect. We see in Figure 9 that different bandwidths generate very close MdMRE and Pred(25) results for N-ABE methods. More importantly there is no observable pattern in small changes due to kernel or bandwidth alterations. Another property of Nasa93 results is that uniform weighting in ABE0 gains higher estimation accuracies (lower MdMRE and higher Pred(25) values) than corresponding N-ABE methods. Lastly, we need to mention that similar to our use of Cocomo81 dataset, for Nasa93 we use all 93 instances and all 17 features in our treatments as well.

4.1.3 Results for Desharnais

We provide the MdMRE and Pred(25) values for Desharnais dataset in Figure 10. Among all kernel-bandwidth combinations we do not see a case where use of nonuniform weighting improves the performance of uniform weighting. Therefore, particular characteristic of previous datasets in terms of being indifferent to different kernel methods is also valid for Desharnais dataset. Furthermore, what we see from Figure 10 is that instead of improving ABE0 methods, N-ABE methods generate considerably worse MdMRE and Pred(25) results. In the experiments we used 81 instances from Desharnais dataset. For missing attributes of 4 projects in Desharnais, simple mean imputation is employed [?].

4.1.4 IRWM Results for All Datasets

Figure 11 presents our last table for MdMRE and Pred(25) results. The difference between the previous MdMRE-Pred(25) results and the ones in Figure 11 is that previous results belong to a N-ABE methods in which non-uniform weighting was done via standard kernel methods (minimum human interaction), whereas results in Figure 11 belong to a N-ABE method whose weighting strategy is defined by human experts (complete human dependence). The weighting strategies between previous figures and Figure 11 are obviously different. However, the trend in the results are very alike, i.e. in none of the 3 datasets can N-ABE methods outperform ABE0.

4.2 Evaluation of WIN-TIE-LOSS Results

In this section we evaluate the performance of non-uniform weighting and uniform weighting according to our third evaluation criterion that is called win-tie-loss calculation. Since we have 10 settings for each kernel subject to 20 runs, the sum of win, tie and loss values can be at most 180 ((10 settings - 1 setting itself) * 20 runs = 180 ).

4.2.1 Results for Cocomo81

In Figure 12 the win-tie-loss values for Cocomo81 are given. The first observation we can make from Figure 12 is that smaller number of analogies have always attained higher win values and lower loss values. In other words, in all treatments k = 3 attains the highest win and the lowest loss values.

Remember that the total sum of win-tie-loss values for a single treatment can be at most 180. For all settings, the tie values are most of the time less than 45 (less than 25% of all the comparisons), which means that in 75% or more of the comparisons there is a statistical difference between the methods. Furthermore, when we mutually compare the results of ABE0 with N-ABE methods for a single k value, we see that for none of the k values non-uniform weighting through kernel density estimation improves the win values.

From Figure 12 we can also see the effect of applying different kernels and different bandwidths on the performance of N-ABE. In terms of kernels, we can say that there is not a considerable performance difference between different types. Note that our results are consistent with prior research, which reports that different kernels yield similar results [?]. For Cocomo81 dataset we observe that the same fact is also valid for software effort estimation data. On the other hand, bandwidth was reported to be influential in different contexts [?, ?, ?]. However, we are unable to observe the considerable effect of different bandwidths for Cocomo81 dataset. Therefore, we can propose that software effort data behaves differently when compared to other data types, i.e. unlike the performance on spatial data performance on software effort data is not improved by changes in bandwidth.

4.2.2 Results for Nasa93

Figure 13 shows the win-tie-loss results for Nasa93 dataset. The results for Nasa93 are extremely similar to Cocomo81, that is in all cases the highest win values belong to k = 3 and tie values are usually around 25% of 180 comparisons. Furthermore, application of different kernels for N-ABE does not yield a considerable difference. For instance, for the treatment k = 3 and h = 1/sqrt(size) the difference between the highest and the lowest win value (141 and 122 respectively) is only 19, which is around 10% of all 180 comparisons. Similar to the effect of changing kernels, changing bandwidth also falls short of providing any noticable increase or decrease in estimation performance. Furthermore, we need to point out hat in none of the k values have N-ABE methods provided any improvement in estimation accuracy. This shows us that like Cocomo81 dataset, Nasa93 dataset does not favor non-uniform weighting over uniform weighting.

4.2.3 Results for Desharnais

The win-tie-loss values for our last dataset are given in Figure 14. The interpretation of Figure 14 shows us a similar scenario to previous two datasets: Highest win values were attained by k = 3 and the treatments are statistically different from one another for most of the cases. Furthermore, just like the Cocomo81 and Nasa93 datasets, the effect of different kernels as well as the effect of various bandwidths are negligible and do not follow a certain pattern. Another similarity is that in none of the kernel-bandwidth combinations has N-ABE yielded higher estimation performance than ABE0.

4.2.4 IRWM Results for All Datasets

Up to this point we have observed 315 different settings and saw that neither kernel nor the bandwidth change does have a considerable impact on the performance of N-ABE. Furthermore, we found out that simple ABE0 approach yields higher performance measures in terms of win-tie-loss values. However, standard kernels are not the only alternatives of non-uniform weighting. Another N-ABE methods we use in this research is so called IRWM [?,?]. The win-tie-loss values of all 3 datasets for IRWM weighted N-ABE are given in Figure 15. Since IRWM is a different weighting approach than kernel density estimation, we do not have bandwidths to compare in that scenario. Our reading from Figure 15 is that for none of the three dataset do N-ABE methods outperform ABE0. In other words, just like the kernel weighted N-ABE, IRWM weighted N-ABE also fails to outperform the ABE0 performance. Therefore, in a total of 330 settings we see that N-ABE methods are unable to outperform the performance of simple ABE0 approach.

5 Discussion

Rigorous experimentation presented in this research shows that our results are mostly negative, i.e. different variants of kernel estimation do not increase estimation accuracy. In theory, it seems that the selection of the right kernel could significantly improve effort estimation: For example, an intelligent selection of the kernel might compensate for data scarcity. Other effort estimation researchers have shared this theoretical point of view [?,?]. However, to the best of our knowledge, the effect of selecting different kernel methods has not been rigorously explored in the SE literature. Our pre-experimental intuition in this research was that non-uniform weighting could end up performing no better and perhaps even worse than simple uniform weighting for effort estimation datasets. To demonstrate why we had such a negative opinion, we provide an intuitive example in Figure 16. Assume that for a particular test project, 3 analogies are chosen (P_{1}, P_{2}, P_{3}) with effort values of 10.0, 20.0 and 60.0. Also assume that kernel density estimation assigned the the probabilities of 0.1, 0.3 and 0.5 to these analogies respectively. When we normalize these probability values to a 0-1 interval, the weights assigned to P_{1}, P_{2}, P_{3} become 0.0, 0.5 and 1.0. As we can see in Figure 16, the estimation (i.e. median of analogies) for uniform weighting case would be 20.0, whereas the estimation for non-uniform weighting case would be 35.0. A shift from an estimate of 20.0 to 35.0 is a dramatic change of 75% rather than a slight correction. Therefore, our intuition was that non-uniform weighting in software effort estimation could be disruptive rather than constructive.

Then the most likely question to be raised is “Why do other fields [?, ?, ?] benefit from weighting, whereas effort estimation does not?”. Our belief is that the answer is partially hidden behind the low sample sizes of effort datasets. Scarcity of the samples means that the weighting observes a signal being broadcast from very small number of points in the neighborhood. Although we can tune the size of the neighborhood with the bandwidth value, in low sample sizes kernel estimation still performs poorly.

We can observe the effect of instance size and bandwidth value on the estimation accuracy in Figure 17. In Figure 17 we simulate 50, 100 and 1000 samples coming from two Gaussian probability distribution functions (PDFs): N(20, 5) and N(35, 5). Then we use kernel density estimation technique with a Gaussian kernel to estimate the density at discrete values of x in [0-55] interval with a step-size of 1. In Figure 17 the x-axis shows the instance IDs for discrete x values and instance IDs are equal to actual x values. An x with the ID of say 10 corresponds to the discrete x value of 10. The y-axis in Figure 17 shows the probability values for each x instance. To better observe the probability values, we plot data as logarithmic scale for y-axis while still reporting the actual probability values rather than the logged values. The probability values of x for the actual Normal distributions are shown with a solid line. Ideally we want to get estimates as close as possible to the envelope indicated by the solid line. The density estimates with different bandwidths (different h values) are shown with different lines in Figure 17. Three bandwidth values mean different neighborhood sizes and they enable different number of sample points to fall into neighborhoods. h = 0.001 is an example of a too small bandwidht value. As we see, for all sample sizes h = 0.001 does hardly let any points to fall into the neighborhood of x, therefore the probability is zero most of the time and whenever point(s) fall into neighborhood of x, we observe sparks of probability. Contrary to 0.001, bandwidth value of 10 is a too large value and it cannot model the underlying Gaussian distributions either. When we look at Figure 17, we see that estimations made with h = 10 are over-smoothed and h = 10 can only estimate a single Gaussian distribution instead of two. In this simple example h = 1 yields the closest estimates to the actual probability values. However, we see that even for h = 1, the sample size plays a critical role in estimation. When we look at the fit between h = 1 estimates and the actual PDFs in Figure 17, we see that as we increase the size of simulated points the large deviations that we observe in Figure 17(a) diminishes through Figure 17(c).

In case of signal processing the sample sizes are closer to Figure 17(c) and as we see from the simulation example, kernel estimates can successfully model such densely populated datasets. In this simulation example we chose the instance sizes of 50 and 100 on purpose, because, software effort datasets used in our research fall into the range of these numbers. Hence Figure 17(a) and Figure 17(b) can give us hints about the kernel density estimation applied on software effort estimates. When we observe behaviour of kernel estimates for low sample sizes in Figure 17(a) and Figure 17(b), it is somewhat expected to observe lower performance values in sparsely populated datasets like software effort datasets.

Although simulation study also confirms our intuitive feeling, data coming from simulation can hardly model the real-life data in software effort estimation domain and we wanted to observe whether results of intuitive examples would also hold for real-life datasets.

6 Threats to Validity

We will address the threats to validity of this research under 3 categories: Internal validity, external validity and construct validity. Before addressing our research in terms of these categories of threats to validity, we would like to give their conscise definitions.

– Internal validity asks to what extent the cause-effect relationship between dependent and independent variables holds [?].

– External validity questions the ability to generalize the results [?].

– Construct validity (i.e. face validity) makes sure that we in fact measure what we intend to measure [?].

The perfect case for the satisfaction of internal validity would be the application of a theory that was learned from past experiences to new situations. However, data in software effort estimation domain is a relatively sparse resource and most of the studies make use of commonly-explored datasets like the ones we use in this research. Therefore, the issue of internal validity thereatens all effort studies that use past data. However, we can mitigate this threat by simulating the behavior of a learned theory in new settings. In our study, we utilize leave-one-out method for all treatments to address such internal validity issues. Leave-one-out selection enables us to separate the training and test sets completely in each experiment, thereby making the test sets completely new situations for the training sets.

To observe the generalizability of our results, we perform extensive experiments on 3 datasets. The datasets are widely used in software effort estimation community and have very diffferent characteristics in terms of various criteria such as size, number of features, types of features and measurement method. Furthermore datasets are subject to rigorous experimentation where we investigate the effects of non-uniform weighting on performance under 330 settings. Our observations for all the settings are extremely similar. Therefore, for the datasets used in our research, our opinion is that the results have external validity. However, to have full confidence in our claims when saying that N-ABE methods fail to outperform ABE0, our study needs to be replicated on other dataset and possibly with different non-uniform weighting strategies.

The choice of performance measures is an open issue in software effort estimation domain. For example MdMRE is recognized as de facto evaluation criterion for cost estimation models [?] and it appears as a practical performance evaluation option to a number of researchers [?, ?, ?]. On the other hand, use MdMRE is still criticized for being unreliable [?, ?]. Foss et. al. for instance shows that MRE can be misleding, if used as the only performance criterion [?]. Therefore, a study willing to have construct validity should not merely rely on MRE-based measures. To measure what we really intend to measure, we make use of win-tie-loss measures apart from MdMRE and Pred(25). Of course all these criteria have their inherent weaknesses and strenghts. Our aim in combining these measures is to use them in a complementary manner. For example strenght of MdMRE as well as Pred(25) is that they give a general picture of per instance-based evaluation. However, median error measure and Pred(25) become too general when averaged over 20 runs and we do not have much knowledge regarding individual runs. Win-tie-loss measures on the other hand compare each method with one-another for each run and allow us to have a better opinion concerning individual runs. Furthermore, MdMRE and Pred(25) show us the difference between two methods based on MRE. But this difference may not always have statistical significance. To ensure the statistical validity of our results we make use of Mann-Whitney U test at a significance level of 95% in win-tie-loss calculations. Therefore, our use of different performance measures such as MdMRE, Pred(25) as well as win-tie-loss, provides different perspectives of the results and lets us know if the results have statistical significance.

7 Conclusions

In this research we tried kernel density estimation as a non-uniform weighting strategy for ABE. We conducted extensive experiments with various kernels and observed the performance variations between U-ABE and N-ABE methods. For the datasets used in our research (Cocomo81, Nasa93 and Desharnais) there was not a single case where N-ABE methods outperformed ABE0. We hesitate to discourage further research with different experimental settings or with different datasets. However, if similar use of kernels is to be adopted, we do not recommend the use of kernel methods as a weighting strategy in ABE.

Unlike previous studies in different domains that use kernel density estimation and report improved accuracy values [?, ?], we did not observe such an effect on software effort datasets. The reason for different results between previous research and our research may lie in the different characteristics of the datasets used in different studies. For instance none of the previous studies we searched through uses software effort datasets and the used datasets are much more densely populated than software effort datasets.

The literature is doubtful about the ability to make model rankings as offered in this paper between ABE0 and N-ABE [?,?,?]. For example Foss et. al. comments that “...is futile to search for the Holy Grail: A single, simple-to-use, universal goodness-of-fit kind of metric, which can be applied with ease to compare (different methods)” [?]. The evidence offered in literature that such rankings may change is based on:

– the random selection of data,

– the dataset used,

– the evaluation metric used for comparison.

However, a convincing counter example to this scenario would be to show that method A is better than method B over multiple datasets, over multiple random selections of train/test data and over multiple evaluation criteria. In this research:

– we used 3 of the most commonly utilized public datasets,

– we adopted leave-one-out method in which every instance is considered as a test instance once,

– covered 330 different settings,

– evaluated our results w.r.t. to 3 different evaluation criteria that have different inherent assumptions.

7.1 Answers To Research Questions

In this section we map the evaluation of our results to particular research questions that guided us in this research.

RQ1 Is there any evidence that non-uniform weighting improves the performance of ABE? In our experiments we do not see a single case in which non-uniform weighting improved ABE. On the contrary, for all settings ABE0 yields much better results than N-ABE methods. Therefore, the evidence suggests that non-uniform weighting decreases estimation accuracy in ABE systems.

RQ2 What is the effect of different kernels for non-uniform weighting ABE? We observe inconsistent and extremely limited effect due to change of kernels. There are only slight variations in performance when different kernels are used. However, these variations do not follow a definite pattern and they are far from being considerable.

RQ3 What is the effect of different bandwidths when used for non-uniform weighting ABE? Change of bandwidths shows a very limited and random effect on the accuracy values too. Therefore, we cannot say that applying different bandwidths has a certain effect on N-ABE performance.

RQ4 How do the characteristics of software effort datasets influence the performance of kernel weighting in non-uniform weighting ABE? In this paper we reported different results than the previous studies that used kernel estimation on different data types. This fact can be attributed to the particular characteristics of software effort datasets. Effort datasets are much smaller than most of the datasets in different domains. The dependent variable (effort value of a completed project) is highly variable. Furthermore, the attribute values are very open to personal judgment and error. All these factors suggest that non-parametric methods may be failing due to inherent characteristics of software effort data.

RQ5 Which parameters among kernel, bandwidth and selected analogy number matter the most in terms of estimation accuracy? Our results showed that in a N-ABE model with kernel density estimation neither kernel nor the bandwidth matters. However, the selected number of analogies (k) matters greatly. Unlike change of kernel and bandwidth that did not provide any noticable change in terms of accuracy values, k value influenced performance measures considerably and we always observed higher accuracies for lower k values.

8 Future Work

We can identify 3 main domains in which this study may be extended:

1. Dataset: In this research we used 3 common software effort datasets. However, this study may be replicated on other software effort datasets as well. Furthermore, ABE is not restricted to software effort estimation domain. Kernel density estimation may be experimented on other ABE-applicable datasets as well.

2. Weighting Strategy: Strategy for non-uniform weighting in a N-ABE method may be completely different than kernel density estimation. Another future direction can be experimentation on different non-uniform weighting strategies that are preferably based on different assumptions.

3. Kernel Type: We used 5 kernels (including IRWM) as non-uniform weighting strategies in our research. But the ones used in this research are obviously not the only kernel types. It may be the case that other particular kernels will perform differently than the ones used here. Therefore, one future direction to this research would be the investigation of different kernels for better performance.

The experiments shown in this research took three months to research, design, execute, then write up. It turns out that we could have spent the time more productively on other issues. Our pre-experimental intuition that “non-uniform weighting in the data sparse domain of software effort estimation could be disruptive rather than constructive” turned out to be correct. We suggest researchers willing to follow future directions to bear these facts in mind as well.

(a) Uniform Kernel  (b) Triangular Kernel

(c) Epanechnikov Kernel  (d) Gaussian Kernel

(e) IRWM Kernel for k=10

Fig. 3: General shapes of the kernel types used in this research. Note that IRWM is not symmetric unlike other kernels, that is due to the fact IRWM has a different weighting procedure than other kernels (see Equation ??).

(a) h=0.1  (b) h=0.05

(c) h=0.02  (d) h=0.005

Fig. 4: We see the effect of bandwidth on kernel density estimation. From Figure 4(a) to Figure 4(d), the bandwidth gets smaller and smaller and we observe a transition from over-smoothing to under-smoothing.

Fig. 5: In the case of ABE0 all instances are given equal probability values, hence equal weights. However, uniform kernel prefers some instances over the others: Only a certain portion of the instances are given equal weights.

Dataset  Features T = |P rojects| Content  Units
Cocomo81  17  63 NASA projects  months
Nasa93  17  93 NASA projects  months
Desharnais  12  81 Canadian software projects hours

Total: 237

Fig. 6: We used 237 projects coming from 3 datasets. Datasets have different characteristics in terms of the number of attributes as well as the measures of these attributes.

win_{i} = 0, tie_{i} = 0, loss_{i} = 0 win_{j} = 0, tie_{j} = 0, loss_{j} = 0 if MANN-WHITNEY(MRE′s_{i}, MRE′s_{i}) says they are the same then

tie_{i} = tie_{i} + 1;

tie_{j} = tie_{j} + 1; else

if median(MRE′s_{i}) < median(MRE′s_{j}) then win_{i} = win_{i} + 1 loss_{j} = loss_{j} + 1

else win_{j} = win_{j} + 1 loss_{i} = loss_{i} + 1

end if end if

Fig. 7: Pseudocode for Win-Tie-Loss Calculation Between Method i and j

h=1/sqrt(size)  h=2  h=4  h=8  h=16

k  MdMRE Pred(25)%MdMREPred(25)% MdMREPred(25)% MdMREPred(25)% MdMREPred(25)%
Uniform 3  0.33  8.65  0.33 8.25 0.35 9.84 0.31 9.37 0.40 7.78
5  0.35  7.94  0.36 7.62 0.39 8.17 0.35 8.97 0.43 8.49
7  0.37  8.41  0.39 10.32 0.44 7.46 0.40 9.05 0.45 9.13
9  0.43  7.86  0.44 8.57 0.47 8.89 0.46 9.29 0.49 9.92
d  0.79  12.30  0.82 7.62 0.78 7.94 0.62 8.25 0.61 11.75
3+N 0.82  3.25  0.77 5.00 0.79 7.62 0.78 5.32 0.79 6.43
5+N 0.90  5.40  0.87 5.79 0.88 5.48 0.87 6.03 0.88 5.63
7+N 0.92  6.03  0.91 6.51 0.92 6.19 0.91 5.95 0.92 7.22
9+N 0.93  7.30  0.93 5.16 0.94 5.48 0.93 5.40 0.94 6.98
d+N 0.94  9.21  0.97 5.79 0.98 6.59 0.97 6.19 0.96 6.67
Triangular 3  0.33  9.92  0.28 8.25 0.33 8.57 0.36 8.57 0.33 8.17
5  0.36  8.73  0.35 9.44 0.36 8.33 0.39 9.37 0.39 9.44
7  0.40  7.78  0.39 8.49 0.40 7.30 0.43 8.49 0.44 8.10
9  0.43  8.49  0.45 9.13 0.45 8.17 0.47 9.37 0.49 8.81
d  0.82  9.92  0.69 12.78 0.63 7.30 0.54 12.86 0.68 12.70
3+N 0.82  2.06  0.72 5.08 0.73 3.97 0.76 5.32 0.76 4.84
5+N 0.90  4.44  0.73 7.62 0.75 8.25 0.78 6.59 0.78 6.90
7+N 0.92  6.19  0.74 8.17 0.75 7.78 0.78 7.70 0.77 6.03
9+N 0.94  5.71  0.74 8.73 0.75 7.54 0.77 8.49 0.77 7.54
d+N 0.92  2.06  0.78 12.70 0.76 7.78 0.77 8.57 0.77 11.51

Epanechnikov

3  0.38  8.65  0.25 8.02 0.30 9.29 0.33 9.68 0.28 9.05
5  0.41  9.76  0.33 8.41 0.35 8.81 0.36 8.89 0.33 9.13
7  0.44  9.84  0.36 8.97 0.37 8.41 0.40 9.76 0.38 8.49
9  0.50  8.81  0.43 7.86 0.43 8.81 0.46 10.24 0.44 9.13
d  0.56  12.22  0.44 7.38 0.60 12.14 0.50 10.24 0.51 9.44
3+N 0.84  2.86  0.66 5.87 0.68 6.75 0.69 5.48 0.68 5.63
5+N 0.92  2.86  0.65 8.17 0.67 8.49 0.68 8.73 0.67 6.98
7+N 0.93  4.29  0.64 9.13 0.67 6.59 0.68 6.75 0.66 7.06
9+N 0.94  5.95  0.65 9.13 0.66 6.67 0.67 9.21 0.65 9.21
d+N 0.94  8.49  0.65 9.92 0.69 11.35 0.68 9.21 0.67 11.19
Gaussian 3  0.42  9.52  0.33 9.60 0.33 9.84 0.33 9.13 0.41 8.97
5  0.43  8.65  0.36 9.37 0.37 10.63 0.36 9.37 0.43 10.32
7  0.46  9.52  0.39 8.65 0.40 8.33 0.40 9.84 0.47 10.00
9  0.49  8.81  0.44 8.57 0.44 9.60 0.45 8.65 0.50 8.97
d  0.59  11.90  0.66 11.51 0.63 11.59 0.60 13.49 0.61 10.32
3+N 0.84  2.38  0.69 5.71 0.71 5.87 0.70 5.08 0.73 6.98
5+N 0.92  3.57  0.68 7.22 0.70 6.75 0.68 8.10 0.71 6.27
7+N 0.94  5.56  0.67 7.06 0.68 6.59 0.66 7.46 0.70 6.98
9+N 0.95  5.32  0.67 7.22 0.67 8.02 0.66 8.17 0.70 8.17
d+N 0.95  9.92  0.71 7.78 0.70 9.52 0.69 7.46 0.71 10.79

Fig. 8: MdMRE and Pred(25) results for Cocomo81 dataset. The column k lists the k values. +N stands for non-uniform weighting. Cocomo81 results confirm that: 1) Neither the bandwidth nor the kernel type have a significant effect on the performance and 2) N-ABE methods cannot outperform the U-ABE method of ABE0.

h=1/sqrt(size)  h=2  h=4  h=8  h=16

k  MdMRE Pred(25)%MdMREPred(25)% MdMREPred(25)% MdMREPred(25)% MdMREPred(25)%
Uniform 3  0.40  14.35  0.23 14.19 0.20 14.95 0.43 14.95 0.43 12.96
5  0.43  13.28  0.26 12.58 0.21 13.17 0.43 11.45 0.44 12.96
7  0.43  12.90  0.33 12.10 0.25 13.28 0.43 11.99 0.44 11.40
9  0.43  10.86  0.35 10.54 0.29 11.56 0.43 11.67 0.44 9.57
d  0.81  9.68  0.30 8.12 0.42 11.02 0.43 11.83 0.49 10.38
3+N 0.83  3.92  0.77 8.92 0.75 9.62 0.78 7.69 0.79 7.15
5+N 0.90  3.12  0.86 6.88 0.85 7.53 0.87 6.72 0.88 5.27
7+N 0.93  3.28  0.90 4.68 0.89 5.97 0.91 5.00 0.91 5.00
9+N 0.94  4.25  0.92 5.38 0.92 5.43 0.93 5.48 0.93 2.69
d+N 0.90  11.45  0.83 1.40 0.97 4.09 0.93 3.23 0.97 95.55
Triangular 3  0.32  13.66  0.30 13.76 0.45 12.90 0.29 14.35 0.30 14.35
5  0.40  11.72  0.40 12.47 0.46 12.42 0.31 12.69 0.40 12.37
7  0.43  12.20  0.40 11.72 0.47 11.88 0.37 12.74 0.41 12.58
9  0.42  10.32  0.40 11.83 0.47 11.67 0.37 11.18 0.40 11.24
d  0.32  11.29  0.40 8.92 0.50 8.17 0.31 11.77 0.44 12.58
3+N 0.83  3.23  0.76 6.34 0.79 5.91 0.78 5.92 0.80 6.88
5+N 0.90  2.20  0.77 9.84 0.78 8.01 0.77 8.92 0.78 8.60
7+N 0.92  3.17  0.76 9.95 0.77 9.78 0.76 10.32 0.78 8.98
9+N 0.94  4.14  0.75 11.24 0.76 10.32 0.75 10.43 0.77 10.48
d+N 0.83  2.47  0.74 8.23 0.73 8.28 0.77 11.40 0.74 8.98

Epanechnikov

3  0.29  12.80  0.34 13.49 0.45 13.92 0.45 14.57 0.48 14.09
5  0.39  13.17  0.40 12.85 0.45 12.69 0.46 13.01 0.48 12.96
7  0.39  12.15  0.41 13.49 0.46 12.20 0.47 13.49 0.49 12.80
9  0.39  10.91  0.40 12.42 0.46 12.15 0.47 12.47 0.49 11.08
d  0.40  11.13  0.57 13.49 0.45 11.67 0.47 9.41 0.62 11.08
3+N 0.83  3.06  0.74 7.90 0.76 7.96 0.75 7.20 0.80 7.15
5+N 0.90  2.58  0.71 9.25 0.74 10.65 0.72 10.54 0.75 10.91
7+N 0.93  2.80  0.68 11.45 0.72 9.78 0.70 11.61 0.73 9.89
9+N 0.95  2.74  0.67 11.88 0.70 10.91 0.68 11.51 0.71 11.13
d+N 0.96  5.48  0.66 11.45 0.68 10.65 0.69 9.89 0.68 12.63
Gaussian 3  0.29  13.76  0.41 14.03 0.43 13.17 0.25 14.30 0.46 13.01
5  0.37  13.55  0.41 12.80 0.43 12.20 0.28 12.96 0.47 13.01
7  0.38  14.41  0.44 13.98 0.44 12.31 0.32 12.85 0.49 13.06
9  0.38  11.24  0.42 12.26 0.43 11.13 0.33 10.81 0.49 12.15
d  0.36  9.09  0.67 10.22 0.48 11.83 0.30 10.32 0.51 9.52
3+N 0.83  3.39  0.78 8.39 0.69 6.40 0.72 6.61 0.78 6.83
5+N 0.90  2.74  0.76 10.91 0.67 10.00 0.70 10.70 0.74 11.77
7+N 0.93  3.87  0.73 11.18 0.66 9.95 0.69 10.70 0.72 11.34
9+N 0.94  3.60  0.71 11.29 0.65 10.75 0.67 11.24 0.70 11.29
d+N 0.93  9.73  0.70 9.89 0.63 10.81 0.71 9.46 0.66 9.62

Fig. 9: MdMRE and Pred(25) results for Nasa93 dataset. Neither change of kernel nor the change of bandwidth generates a considerable difference in results. Furthermore, small changes in MdMRE and Pred(25) values due to different kernel-bandwidth combinations do not follow a certain pattern. Another conclusion from this figure is that N-ABE methods fail to improve ABE0.

h=1/sqrt(size)  h=2  h=4  h=8  h=16

k  MdMRE Pred(25)%MdMREPred(25)% MdMREPred(25)% MdMREPred(25)% MdMREPred(25)%
Uniform 3  0.39  34.14  0.40 35.43 0.40 35.12 0.40 34.69 0.37 35.62
5  0.36  37.96  0.37 38.70 0.36 37.16 0.37 36.98 0.35 38.46
7  0.36  36.91  0.37 38.52 0.36 36.85 0.37 36.67 0.36 37.47
9  0.37  35.74  0.37 36.42 0.37 34.94 0.37 35.37 0.36 36.05
d  0.37  34.88  0.36 31.67 0.37 32.84 0.36 31.98 0.36 33.58
3+N 0.81  2.53  0.70 5.86 0.69 5.19 0.70 5.12 0.69 5.31
5+N 0.86  0.93  0.82 1.30 0.81 2.28 0.82 1.79 0.81 2.22
7+N 0.89  0.37  0.87 0.19 0.86 0.12 0.87 0.19 0.86 0.12
9+N 0.91  0.19  0.89 0.00 0.89 0.00 0.90 0.00 0.89 0.00
d+N 0.93  0.12  0.88 0.00 0.96 0.00 0.94 0.00 0.89 0.00
Triangular 3  0.38  36.05  0.36 34.81 0.38 35.49 0.36 34.88 0.39 34.07
5  0.36  38.27  0.35 37.35 0.36 38.40 0.36 37.65 0.36 37.10
7  0.36  37.10  0.36 37.35 0.36 38.15 0.36 37.16 0.36 36.54
9  0.36  35.93  0.36 36.17 0.36 34.44 0.36 36.42 0.37 35.00
d  0.41  32.59  0.36 33.33 0.37 34.44 0.47 35.43 0.36 32.84
3+N 0.44  25.80  0.62 10.06 0.61 9.94 0.61 10.31 0.60 11.54
5+N 0.41  30.93  0.55 14.75 0.54 15.06 0.53 16.05 0.54 16.73
7+N 0.39  34.75  0.50 17.35 0.51 17.35 0.50 19.26 0.51 18.95
9+N 0.38  34.38  0.48 20.74 0.49 20.37 0.48 22.47 0.49 21.54
d+N 0.41  31.31  0.42 25.00 0.39 20.37 0.46 16.79 0.47 30.43

Epanechnikov

3  0.36  34.94  0.40 37.59 0.39 35.43 0.36 35.62 0.38 35.56
5  0.36  37.96  0.34 38.64 0.36 37.28 0.35 37.47 0.36 37.72
7  0.36  35.74  0.36 37.10 0.36 36.73 0.36 36.98 0.36 36.98
9  0.36  34.51  0.36 36.85 0.36 35.49 0.37 34.94 0.36 35.68
d  0.36  32.59  0.36 32.41 0.37 36.73 0.35 31.91 0.40 30.62
3+N 0.80  0.80  0.58 10.93 0.58 12.16 0.58 10.93 0.57 11.98
5+N 0.87  0.31  0.50 17.35 0.50 17.10 0.49 17.41 0.51 16.67
7+N 0.90  0.00  0.46 23.15 0.47 21.85 0.46 22.65 0.47 20.93
9+N 0.92  0.12  0.45 24.26 0.45 24.14 0.44 24.57 0.44 23.70
d+N 0.92  0.12  0.42 30.43 0.38 21.85 0.49 30.00 0.41 30.43
Gaussian 3  0.39  34.01  0.37 35.06 0.40 37.35 0.40 35.25 0.39 35.43
5  0.36  37.90  0.36 38.02 0.39 38.70 0.36 37.72 0.36 38.21
7  0.36  36.98  0.37 36.67 0.39 37.53 0.37 37.10 0.36 37.41
9  0.36  34.81  0.37 34.88 0.37 36.05 0.37 33.77 0.37 35.19
d  0.35  33.89  0.36 31.54 0.39 34.81 0.36 31.60 0.36 38.21
3+N 0.80  1.73  0.60 11.42 0.61 10.68 0.60 11.23 0.58 12.35
5+N 0.86  0.19  0.51 17.78 0.52 16.36 0.50 17.22 0.50 18.64
7+N 0.90  0.06  0.47 22.41 0.49 20.12 0.46 21.30 0.45 23.15
9+N 0.91  0.00  0.45 24.81 0.47 23.64 0.45 23.77 0.44 25.80
d+N 0.93  0.93  0.47 30.00 0.49 25.68 0.43 30.42 0.38 18.64

Fig. 10: MdMRE and Pred(25) results for Desharnais dataset. None of the different kernel-bandwidth combinations can improve the performance of N-ABE methods to a point where they outperform the U-ABE method of ABE0.

Cocomo81  Nasa93  Desharnais
k  MdMRE Pred(25)% MdMRE Pred(25)% MdMRE Pred(25)%
IRWM 3  0.42  8.97  0.40  36.23  0.39  35.93
5  0.44  8.81  0.42  38.2  0.37  37.59
7  0.48  8.57  0.43  41.82  0.37  38.52
9  0.51  9.05  0.43  45.49  0.36  35.43
d  0.86  12.62  0.80  179.44  0.37  35.43
3+N  0.59  8.02  0.57  55.87  0.46  24.01
5+N  0.64  8.33  0.61  59.91  0.49  19.07
7+N  0.66  7.3  0.63  61.75  0.51  16.79
9+N  0.68  7.62  0.64  63.02  0.51  16.54
d+N  0.79  9.68  0.74  95.44  0.51  16.54

Fig. 11: MdMRE and Pred(25) results of Cocomo81, Nasa93 and Desharnais for IRWM weighted N-ABE. k stands for the number of analogies used for estimation and +N sign means that non-uniform weighting is used for estimation. Similar to N-ABE methods through standard kernels, expert proposed kernel cannot perform an improvement to ABE0.

h=1/sqrt(size)  h=2  h=4  h=8  h=16
k  WIN TIE LOSS WIN TIE LOSS WIN TIE LOSS WIN TIE LOSS WIN TIE LOSS
Uniform 3  144 32  4  144 32  4  144 32  4  144 32  4  144 32  4
5  127 48  5  127 48  5  127 48  5  127 48  5  127 48  5
7  116 48  16  116 48 16 116 48 16 116 48 16 116 48 16
9  113 28  39  113 28 39 113 28 39 113 28 39 113 28 39
d  76 24  80  76 24 80  76 24 80  76 24 80  76 24 80
3+N  74 51  55  74 51 55  74 51 55  74 51 55  74 51 55
5+N  28 34 118  28 34 118 28 34 118 28 34 118 28 34 118
7+N  12 41 127  12 41 127 12 41 127 12 41 127 12 41 127
9+N  6  34 140  6  34 140  6  34 140  6  34 140  6  34 140
d+N 16 32 132  16 32 132 16 32 132 16 32 132 16 32 132
Triangular 3  147 31  2  147 31  2  147 31  2  147 31  2  147 31  2
5  132 46  2  132 46  2  132 46  2  132 46  2  132 46  2
7  123 43  13  123 43 14 123 43 14 123 43 14 123 43 14
9  116 23  40  116 23 41 116 23 41 116 23 41 116 23 41
d  97  2  80  97  2  81  97  2  81  97  2  81  97  2  81
3+N  78 17  85  78 17 85  78 17 85  78 17 85  78 17 85
5+N  54  8  117  54  8 118 54  8 118 54  8 118 54  8 118
7+N  22 23 134  22 23 135 22 23 135 22 23 135 22 23 135
9+N  9  25 145  9  25 146  9  25 146  9  25 146  9  25 146
d+N  0  22 157  0  22 158  0  22 158  0  22 158  0  22 158

Epanechnikov

3  145 33  1  145 33  2  145 33  2  145 33  2  145 33  2
5  139 38  2  139 38  3  139 38  3  139 38  3  139 38  3
7  124 40  15  124 40 16 124 40 16 124 40 16 124 40 16
9  116 18  45  116 18 46 116 18 46 116 18 46 116 18 46
d  97  4  78  97  4  79  97  4  79  97  4  79  97  4  79
3+N  79 16  85  79 16 85  79 16 85  79 16 85  79 16 85
5+N  41 18 120  41 18 121 41 18 121 41 18 121 41 18 121
7+N  10 42 127  10 42 128 10 42 128 10 42 128 10 42 128
9+N  6  40 133  6  40 134  6  40 134  6  40 134  6  40 134
d+N  2  29 148  2  29 149  2  29 149  2  29 149  2  29 149
Gaussian 3  136 42  2  137 32 11 138 38  4  139 36  5  142 32  6
5  130 48  2  129 40 11 133 44  3  131 44  5  132 42  6
7  116 57  7  117 41 22 122 47 11 119 46 15 122 41 17
9  114 33  33  108 19 53 115 25 40 114 26 40 113 22 45
d  95  7  78  78 27 75  88 16 76  70 32 78  95  4  81
3+N  66 34  80  80 60 40  80 24 76  79 40 61  85 21 74
5+N  27 39 114  59 13 108 61  3 116 61 14 105 60  5 115
7+N  7  50 123  41 10 129 40  3 137 39  7 134 38  7 135
9+N  4  53 123  20 10 150 19  4 157 20  6 154 20  7 153
d+N  1  45 134  0  10 170  0  4 176  0  5 175  0  5 175

Fig. 12: Win-tie-loss results for Cocomo81. The N-ABE experiments are shown with a +N sign, whereas the dynamic k is represented by a d. We used 5 different bandwidths for 4 different kernels. Similar to other data types, for Cocomo81 we do not see an improvement coming from different kernels. However, unlike other data types, we are unable to observe an improvement coming from change of bandwidth values too.

h=1/sqrt(size)  h=2  h=4  h=8  h=16
k  WIN TIE LOSS WIN TIE LOSS WIN TIE LOSS WIN TIE LOSS WIN TIE LOSS
Uniform 3  141 39  0  138 42  0  146 34  0  138 42  0  146 34  0
5  135 45  0  127 52  1  130 49  1  129 51  0  133 46  1
7  120 52  8  111 56 13 119 46 15 120 53  7  118 50 12
9  119 32  29  105 50 25 116 36 28 120 39 21 114 35 31
d  100 2  78  100 38 42 100 13 67 100 1  79 100 13 67
3+N  76  4  100  80  0 100 80  0 100 80  0 100 80  0 100
5+N  51 10 119  60  0 120 60  0 120 60  0 120 60  0 120
7+N  27 18 135  40  0 140 40  0 140 40  0 140 40  0 140
9+N  16 15 149  20  0 160 20  0 160 20  0 160 20  0 160
d+N  2  9  169  0  0 180  0  0 180  0  0 180  0  0 180
Triangular 3  122 47  11  119 46 15 125 43 12 128 40 12 110 53 17
5  115 52  13  107 57 16 115 54 11 120 49 11  98 63 19
7  103 60  17  97 58 25 104 57 19 110 49 21  88 61 31
9  99 41  40  91 52 37 104 46 30 109 35 36  83 36 61
d  90 32  58  85 39 56  90 14 76  89  5  86  98 63 19
3+N  91 44  45  71 68 41  50 57 73  55 62 63  77 77 26
5+N  59 12 109  11 59 110 11 50 119  3  60 117  7  73 100
7+N  32 20 128  6  67 107 15 53 112  9  61 110  2  77 101
9+N  16 22 142  10 68 102 19 54 107 12 59 109  9  66 105
d+N  0  16 164  17 58 105 37 32 111 33 44 103  7  73 100

Epanechnikov

3  139 41  0  135 43  2  144 35  1  133 47  0  137 43  0
5  126 54  0  118 61  1  133 47  0  124 55  1  121 58  1
7  122 48  10  108 56 16 122 48 10 112 62  6  111 59 10
9  121 41  18  103 56 21 119 31 30 112 49 19 112 53 15
d  100 0  80  102 52 26  99  4  77 100 25 55 100 25 55
3+N  77  3  100  0  22 158  0  22 158  0  15 165  0  7 173
5+N  48 13 119  16 34 130 16 34 130 15 28 137 16 28 136
7+N  21 24 135  24 44 112 26 42 112 27 32 121 24 36 120
9+N  14 24 142  27 49 104 34 40 106 38 30 112 39 34 107
d+N  2  12 166  42 33 105 39 33 108 61 13 106 57 23 100
Gaussian 3  124 44  12  122 45 13 102 60 18 127 41 12 117 52 11
5  113 54  13  113 54 13  92 70 18 119 50 11 114 56 10
7  97 63  20  103 57 20  86 71 23 108 55 17 105 61 14
9  90 53  37  105 46 29  83 66 31 108 38 34 107 52 21
d  88 42  50  88 13 79  83 61 36  85  8  87  90 13 77
3+N  92 48  40  60 52 68  75 60 45  50 47 83  50 46 84
5+N  55 16 109  7  38 135 20 37 123 13 37 130  8  32 140
7+N  23 33 124  16 50 114 17 56 107 16 48 116 17 49 114
9+N  4  40 136  25 44 111 19 61 100 23 48 109 26 46 108
d+N  0  35 145  44 35 101 24 56 100 48 34 98  47 31 102

Fig. 13: Win-tie-loss results for Nasa93. Results we have for Nasa93 are very similar to Cocomo81 dataset: Neither changing kernels nor the bandwidths provides a noticable change in win-tie-loss values. Also uniform weighting always outperform non-uniform weighting.

h=1/sqrt(size)  h=2  h=4  h=8  h=16
k  WIN TIE LOSS WIN TIE LOSS WIN TIE LOSS WIN TIE LOSS WIN TIE LOSS
Uniform 3  123 55  2  123 57  0  120 60  0  120 59  1  126 54  0
5  124 56  0  121 59  0  118 62  0  119 61  0  121 59  0
7  116 61  3  116 62  2  115 64  1  115 64  1  114 64  2
9  116 53  11  115 56  9  115 59  6  115 59  6  115 50 15
d  101 15  64  100 16 64 100 19 61 101 17 62 101 19 60
3+N  79  1  100  80  0 100 80  0 100 80  0 100 80  0 100
5+N  52  9  119  60  0 120 60  0 120 60  0 120 60  0 120
7+N  26 21 133  40  0 140 40  0 140 40  0 140 40  0 140
9+N  18 16 146  20  0 160 20  0 160 20  0 160 20  0 160
d+N  0  3  177  0  0 180  0  0 180  0  0 180  0  0 180
Triangular 3  120 60  0  122 58  0  122 57  1  114 65  1  112 68  0
5  116 64  0  122 58  0  120 60  0  108 72  0  103 77  0
7  102 76  2  115 63  2  114 65  1  100 79  1  100 76  4
9  101 64  15  111 56 13 104 69  7  100 70 10 100 65 15
d  96 48  36  100 25 55 101 27 52 100 70 10 104 76  0
3+N  0  0  180  2  34 144  0  44 136  0  46 134  0  47 133
5+N  20 15 145  3  53 124  5  59 116  3  65 112  8  66 106
7+N  33 50  97  14 53 113 12 62 106 11 63 106 16 64 100
9+N  36 49  95  23 54 103 17 61 102 19 61 100 26 53 101
d+N 39 48  93  42 38 100 27 52 101 19 61 100  3  64 113

Epanechnikov

3  132 48  0  130 50  0  126 54  0  123 57  0  123 57  0
5  118 61  1  126 53  1  118 61  1  120 60  0  123 57  0
7  116 56  8  119 59  2  105 73  2  114 65  1  117 60  3
9  114 57  9  118 49 13 100 68 12 114 53 13 110 61  9
d  103 12  65  93 10 77 100 46 34 100 23 57 100 19 61
3+N  80  0  100  1  26 153  0  21 159  0  12 168  0  18 162
5+N  57  2  121  9  41 130 10 43 127 12 36 132 11 36 133
7+N  34  8  138  23 42 115 18 59 103 23 42 115 21 49 110
9+N  20  8  152  34 35 111 29 51 100 35 36 109 30 46 104
d+N  0  0  180  49 31 100 32 48 100 54 26 100 47 33 100
Gaussian 3  121 59  0  122 58  0  126 54  0  126 54  0  123 57  0
5  121 59  0  115 65  0  120 60  0  119 61  0  115 64  1
7  117 62  1  113 65  2  118 59  3  113 65  2  113 65  2
9  115 55  10  109 63  8  118 53  9  113 55 12 108 65  7
d  100 17  63  101 29 50  98 12 70 100 23 57 102 27 51
3+N  80  0  100  0  17 163  0  21 159  0  19 161  0  14 166
5+N  54  4  122  10 30 140 10 44 126 10 38 132 10 36 134
7+N  35 11 134  25 36 119 20 48 112 17 45 118 23 43 114
9+N  20  7  153  36 36 108 32 36 112 31 37 112 33 40 107
d+N  0  0  180  59 21 100 50 29 101 65 15 100 55 25 100

Fig. 14: Win-tie-loss results for Desharnais. The implications we have observed in Cocomo81 and Nasa93 repeat for Desharnais dataset: Change of kernels does not provide a significant change in win-tie-loss values and neither does the change of bandwidths. There are only some small changes in different kernel-bandwidth combinations but we can not observe a pattern. Furthermore, ABE0 has a better estimation performance than all N-ABE methods.

Cocomo81  Nasa93  Desharnais
k  WIN TIE LOSS WIN TIE LOSS WIN TIE LOSS
IRWM 3  143 37  0  141 39  0  126 54  0
5  128 50  2  126 54  0  120 60  0
7  115 55 10 115 53 12 115 62  3
9  101 46 33 117 43 20 116 55  9
d  0  56 124 97 16 67 101 13 66
3+N  88 25 67  78  4  98  80  0 100
5+N  49 48 83  49 14 117 50 10 120
7+N  28 59 93  22 29 129 24 25 131
9+N  23 52 105 19 19 142 16 19 145
d+N  0  22 158  0  1 179  0  6 174

Fig. 15: Win-tie-loss results of Cocomo81, Nasa93 and Desharnais for IRWM weighted N-ABE. The notation in this figure is similar to previous figures: Non-uniform weighting is represented by a +N sign and dynamic kernel is represented by d. IRWM is a different weighting strategy than standard kernel weighting, hence we do not see bandwidth information in this figure. Results are similar to previous scenarios: Lower k values attain higher win values and lower loss values. Furhermore, non-uniform weighting is unable to outperform uniform weighting.

Analogies  P_{1}  P_{2}  P_{3}
Effort Values  10.0 20.0 60.0
Probabilities  0.1  0.3  0.5
Weights  0.0  0.5  1.0
Uniform Weighting Estimate  20.0
Non-Uniform Weighting Estimate  35.0

Fig. 16: An intuitive example. In a 3 analogy case, there is a 75% change for a hypothetical test project between uniform and non-uniform weighting effort estimates.

(a) 50 Sample Points: Note the bad fit due to low sample size.

(b) 100 Sample Points: Note the better fit due to increased sample size.

(c) 1000 Sample Points: Note the optimum fit due to high sample size.

Fig. 17: The effect of sample size and bandwidth on kernel density estimation. The choice of optimum bandwidth (h value) is important. However, even with the optimum bandwidth, one still needs enough number of points for successful kernel density estimation. In this figure the best h value is 1. In terms of sample size, the value of 50 appears to be too small. As we increase the sample size to 100, we get better fit between estimated and actual probability values. But for a very close fit, we need to go up to 1000 sample points.

Replies to reviewer comments

Thank you for your detailed reviews on the previous version. They were most helpful. Replies to comments are below.

Editor

The empirical study produces a negative result - which is generally quite welcome. However, the reviewers (and I) question the generality of your conclusions! Does your result mean that ”Kernel density estimation” must not be investigated any further, or...

We completely agree with your concerns. We made our suggestion/conclusion more clear in this version. The last sentence of Abstract reads:

“Hence, -provided that similar experimental settings are adopted- we discourage the use of kernel methods as a weighting strategy in ABE. ”

Also the last sentence of the 1^{st} paragraph in Conclusion states:

“We hesitate to discourage further research with different experimental settings or with different datasets. However, if similar use of kernels is to be adopted, we do not recommend the use of kernel methods as a weighting strategy in ABE.”

If you think that you can accommodate these concerns (especially the concern of Reviewer 2), please submit a major revision”.

We highly value your invitation for a re-submit after a major revision. Therefore, we paid particular attention to address every comment of reviewers. After following through the invaluable comments of reviewers, this study has become much more sound and to the point. We believe that it is now ready for a re-review. Below is a list of our replies to every individual comment.

Reviewer 1:

The article describes an analysis of a specific type of analogy-based estimation, i.e., kernel-density estimation for non-uniform weights of analogies. The article is very well written and structured. It is easy to read.

Thank you very much, we are very pleased with your comments.

Although the article focuses on an important topic, I have several concerns:

We fully agree with all your concerns. We went through all your comments one by one and restructured our work accordingly. Following is a list of our replies.

- The motivation why kernel-density estimation was chosen as object of study is quite weak and not convincing to me as motivation for performing a comprehensive study. The more obvious reason that the technique could be specifically suited for software engineering projects and data is not addressed.

REPLY HERE!

- The statement ”ABE methods are never enhanced by a particular kernel method” seems to be based on some kind of proof. However, it could also stem from your observations. I recommend to describe in more detail what the (potentially empirical basis) for this statement is.

REPLY HERE!

- Section 2.1 I recommend to also integrate hybrid methods (i.e., methods based on data *and* expert judgment such as CoBRA).

REPLY HERE!

- I also recommend to include some studies that show that the accuracy can be significantly improved by maintaining the estimation model over time. It would be nice for the reader if such kind of concept is also part of your work.

REPLY HERE!

- The so-called ”experiments” are mainly data analyses of existing repositories. This seems to be reasonable to a certain extend. However, it is quite a weak evaluation due to a lack of knowledge about context and assumptions. I recommend for future work to do a customization for a specific company and to derive real problems afterwords.

REPLY HERE!

Overall, the scope (technique, purposes, domain) is very limited and probably only of minor interest for most of the EMSE journal readers.

REPLY HERE!

Reviewer 2:

First of all I would like to applaud the authors’ candor about the negative results rather than trying to dress them up in some way.

Thank you very much for your appreciation.

Nevertheless, I do have some serious problems. Thanks to your and other reviewers’ comments, we restructured our paper and

enriched with further experiments. Below is the list of your comments and our replies.

First, though we need to deal with a potential publication bias against negative results, the work still needs to be interesting.

Show the motivation section (newly written)!

There’s an excellent discussion of these kind of issues in [1]. REPLY HERE!

The situation we have is: ”Another possibility is that a novel hypothesis occurs to a scientist. This scientist proceeds to test. his bright idea only to discover that it is mistaken. Might he or she publish these findings to ward off other scientists who might be tempted to pursue this dead end?” The answer revolves around how interesting the results to which I’d answer not that interesting. My suggestion is a short note of perhaps 20% of the length of this paper.

REPLY HERE!

The second problem is the paper isn’t very well written - it reads like an early draft as it’s often note like or lacks articles etc etc (below I’ve identified a few typos but there are a lot more).

REPLY HERE!

More seriously it’s just hard to follow and a lot of irrelevant information is included eg in the intro why discuss feature selection when the paper is about analogy weighting and selection.

REPLY HERE!

Nor I do get any sense of why analogy weighting is an important and hard problem. Motivation part!

Section 2 is weak eg why have 2 different taxonomies of related work – one will do! REPLY HERE!

The discussion and Eqn 2 (for inverse rank weighting) seems unnecessary – it’s pretty obvious.

REPLY HERE!

Your discussion implies k¿1 since k=1 -¿ U-ABE equiv N-ABE REPLY HERE!

p12 Much as I appreciate the work of Kitchenham and her co-workers this new definition of dataset quality seems desperately over-simplistic.

REPLY HERE!

MMRE? Given the authors are aware of its limitations why not do the statistical tests on absolute residuals? Why pred(25)?

REPLY HERE!

Minor points: p4, l36: problem explicitly addressed by Delphi where individual estimates are kept anonymous.

REPLY HERE!

p6, l22: a a REPLY HERE!

p6, final sentence: meaning is unclear. REPLY HERE!

p7, l27: statistical algorithms? REPLY HERE! p9, l45: started REPLY HERE!

p17, l42: what were the imputed values (so the work could be replicated?) REPLY HERE!

p24, l37: does hardly let?? REPLY HERE!

p26, l46: the choice of performance measure is not an open issue. See Kitchenham et al. [2].

REPLY HERE!

The only issue is what characteristic of accuracy you are actually interested in, normally spread of residuals (error-proneness) and central tendency (bias) would be sufficient.

REPLY HERE!

REFERENCES: [1] H. Browman, ”Negative results,” Marine Ecology Progress Series, vol. 191, pp. 301-309, 1999. [2] B. A. Kitchenham, S. G. MacDonell, L. Pickard, and M. J. Shepperd, ”What accuracy statistics really measure,” IEE Proceedings - Software Engineering, vol. 148, pp. 81-85, 2001.

REPLY HERE!