\section*{Response to Reviewer 1 Comments}

\textbf {Comments (Reviewer 1:) } \\


\textbf{\em 1.1 This paper explores an important question and it looks like a lot of work has been carried out.  BUT it's an early stage.  There's no clear statement of the experimental hypothesis. Much detail is missing, conclusions are scrappy and suffer from drifting all over the place because of the missing experimental hypothesis. There are some gems (I think) if you dig but nobody could replicate the work as it stands.}
\\

The study is not at its early stage, it is an extension of a previous study described in Menzies et al.~\cite{menzies11}.  Menzies et al.~\cite{menzies11} applied 158 predictors to various subsets of two COCOMO datasets, and by changing the random number seeds, performance evaluation criteria and subsets of data, they found the precise ranking of the 158 predictors, the result has been considered {\em stable (or stable conclusion)}. 

The hypothesis in this study is that if we are able to derive a stable conclusion using simulated data, similar behavior should be observed when applying heterogenous datasets from public domains where they are from different sources with various differences in project characteristics.
We have made this statement more explicit in Section 2.2. 

The implementation of the method are available online, one could replicate the entire work described in the paper using both the method and the datasets publicly available. This paper serves the demonstration purpose, the simplicity of the method will ensure its application to more datasets additional to the repository in the near future. 
\\

\textbf{\em
1.2 Ok it's an important question - should we always expect the same conclusions?  I think the authors make a good point about the huge array of different learners, parameters, pre-processors etc  Which matter most?  Classify them and then start to produce testable hypotheses. } \\

We were unable to do that previously, many projects were funded from the 80's. With the data available, we are able to draw a stable conclusion that was previously impossible due to lack of availability of the datasets and the availability of the method combinations. If there are newer methods to be introduced in the near future, the same experiment can be reapplied to examine the newly introduced method against all other existing method combinations in terms of prediction performance accuracy.

For the 2nd question, thank you for your acknowledgment about our use of the array of different combinations, speaking of which matters the most, in our opinion potentially all of them matter. It is impeccable to separate the pre-processors and the learners and to compare their functional difference. But of course one could build a simple model without the use of a pre-processor for the input dataset to the model, in our humble opinion that is not the current focus of the work. 
\\

\textbf{\em
1.3 I think it's unwise to effectively use datasets twice (eg nasa93).  This means some dependence between results.}\\

In our humble opinion, it is necessarily true as that subsets of nasa93 have different properties and collected from different development centers. Essentially they are different datasets. 
\\

\textbf{\em
1.4 If there's a clearer rationale then there will be a justification for some / all of the performance measures. Is the question does it matter which one you choose?  Later on the analysis focuses on Pred(25).  Is this more improtant than the others?  Why?}\\

We treat all the dataset samples and performance measures equally, and we use all of them. It doe not matter which one to choose.

As for Pred(25), it is not important than any others performance measures, it serves the illustrative purpose in this paper only (due to its result range between 0 and 100), for example in Fig 9 and 11. 
\\

\textbf{\em
1.5 The study is in danger of exploring too much.  One problem is some pre-processors eg feature selection has to optimise a particular objective function but elsewhere there are whole set of performance measures.  There seems a potential problem of optimizing on one measure and then using another. }\\

The danger may be overly concerned, in fact this is not a fault of this paper, but rather the purpose of this paper. 
Previously unexplored details matter, a lot. For example we are able to can make a learner bad or good just by changing the pre-processor.

We show that these combinations matters the most and we show that this prompts work on a rich area of potential future work where the technique can be applied.
\\


\textbf{\em
1.6 It is unclear how the datasets were chosen. Are they representative? Is the underlying question how consistently methods work in different situations?}
\\

No strategy was used to choose any particular form of dataset from the PROMISE repository, in fact it contains all the datasets available. 
In 2005, researchers could not handle datasets and PROMISE was established to collect as much datasets as possible from the industry, and the PROMISE is continue to collect data from contributors. The datasets we have used are everything that we could get at that time, and this makes the result more generalizable. 

Ultimately any inductive study is just a reflection of the data used in that study,  all research papers similar to this one suffers from that issue. 
But, in defense of this paper, we say this work suffer much less because we explore many more datasets.
\\

\textbf{\em
1.7 The description of SWR is incomplete and hard to follow.  There are many algorithms and it's unclear how the removal (as opposed to the addition) of features proceeds.}\\

The incomplete description of the SWR which refers to the Step-wise-Regression commonly used in statistics. 
The main approach applied was a standard backward stepwise regression where all candidate variables/features are initially included and testing them one at a time and the ones are irrelevant are to be removed.
The details of the algorithms are not the focus of this paper. 
\\

\textbf{\em
1.8 How does the feature weighting handle categorical features?}
\\

The standard procedures for categorical feature treatment applied. 
For example in the case of Desharnais dataset with ABE0 (Analogy-based Estimation) the dev.env variable has 3 categories. In the actual computation of object distances between project cases, if two comparing objects belong to the same class then the difference is 0, 1 otherwise.   Therefore the feature weightings are either 0 or 1. 
This standard procedure has been described in the well-known study of Shepperd et. al ~\cite{shepperd97} 
\\

\textbf{\em
1.9 Does LReg use all available independent variables? Are categorical features converted to dummy variables?  Details please. It's not that some of these methods are naive (which they are) but because the work cannot be replicated without far more information.}\\ 

Yes, LReg (Linear Regression) uses all the variables, for some variants of regression analysis, only selected features are used based on their statistical significance, commonly described in statistic textbooks. 
To use a categorical variable in a regression analysis we use dummy variables as a commonly used technique described in most of the statistic textbooks. 
\\

\textbf{\em
1.10 Fig 4.  I think you should present the counts (as well as ranks). Later on (Fig 10) shows the need for this. If you were analyzing random data you would still rank a technique as 1, etc.  (ok is in Fig 5 but combining would help).  It seems informally that learner is more important than pre-processor because the ranking tends to group by learner but you don't conclude this in the findings.}
\\

\textbf {******* Can We add counts?*******}
\\

\textbf {\em
1.11 It took a while to get Fig 6 but this is at the heart of your argument so please add more explanation. First threat is your analysis focuses on losses.  Suppose there are many ties (how many?).  It may be your findings just show your procedure cannot discriminate because you are being cautious (insufficient power) using Wilcoxon and many small data sets so most things are viewed as being the same.  The second threat is that if many of your methods are naive then they will always perform badly?  Therefore finding non-foolish methods are consistently better than foolish methods is less significant than appears at first sight?  This needs discussion.} \\

\textbf {******* Need Help guys! *******}
\\


\textbf {\em
1.12 Fig 8/9 Why suddenly focus just on Pred(25).  Is this performance measure more important than the others?  I don't understand why this confirms the 'unique performance' p14} \\

Pred(25) is not important than any others performance measures, it serves the illustrative purpose in this paper only (due to its result range between 0 and 100). 
\\

\textbf {\em
1.13 There is no analysis of the impact of dataset.  This might shed light on the question of using within/ cross company data.}
\\

Figure 6 shows some of the widely used datasets are unable to distinguish and discriminate different learners, which is a totally missed important area of research, for example Albrech and Desharnaise.
\\

\textbf {\em
1.14 The conclusions are hard to follow.
``are not 100 stable, they are only sufficiently stable to draw conclusions`` What's this mean? How do you know?  }
\\

Essentially, we are dealing with a statistical process, based on useful statistics and probabilities to draw assumption and to derive conclusion.
In this case, we have observed enough evidence to draw a valid yet stable conclusion. 
\\


\textbf{\em
1.15 Finding 1 and 2 contradict each other.  Isn't 2 the reason why different researchers disagree?  Also your data seem to suggest method is more important than than preprocessor (Fig 4).  } 
\\

1. Finding 1 and 2 do not contradicting each other, we have rephrased the sentences to make it more clear. 
Finding 1 reports that based on the result, we are seeing that a stable conclusion can be arrived based on the evidence given. 
Finding 2 reports the importance of preprocessor, and we have evidence to show that preprocessor is as important as the learner. 
The purpose of the experiment is trying to identify a stable conclusion for cost estimation methods, i.e. ``Method" =``Preprocessor" + ``Learner"  in this case.
For example, in SWR, the preprocessor is the stepwise feature selection procedure to ensure only the relevant features are included for the next process, the regression or (learner) process, before an estimate is arrived. 
Preprocessor deals with the raw data, has an transformation or filtering effect, the Learner is the actual method itself, the combination of the Preprocessor and the Learner is the Estimation Method. 

2. Finding 2 may explain why some researchers disagree, as the estimates are sensitive to the method combination, therefore the different usage of preprocessor and method combinations will produce different results by different researchers. In here, we are showing that if we have sufficient datasets x preprocessors x learners, we will be able to draw a stable conclusion. Finding 2 shows the variation can be an effect of preprocessor, indeed, not all researchers report the use of preprocessor in their study, it is not surprising that conflicting results have been generated. 

3. Both the learner and preprocessor are important and will have an influence to the prediction outcome, the result do not suggest one is more important than the other (whether it is the learner or preprocessor). The results show that both learner and preprocessor are important.  
\\


\textbf{\em
1.16 Finding 3. It seems that your approach to regression is naive.  Fine because you want to show some general stuff about conclusion stability.  But don't then argue which particular method is best (or worst).  If you do this then a different approach is needed and you need to do each technique properly.
Basically this part suffers badly because you never state clearly what questions your experiment is trying to answer.}
\\

\textbf {******* Need Help guys! *******}
\\


\textbf {\em
1.17 DETAIL
Many unorthodox terms:
I don't think estimator is the best term.  Predictor?  Estimator has a specific meaning in statistics.}
\\

We have changed the term to "Predictor" throughout the paper. Thanks for the suggestion.
\\ 


\textbf{\em
1.18 It's usual to use y as the dependent variable. }
\\

Yes we agree with your suggestion. 
\\

\textbf{\em
1.19 x for number of neighbors (it's overloaded and k is normal)}
\\

Amended, now "\textit{k}NN", thanks. 
\\


\textbf{\em
1.20 T projects?? (p10) }
\\

Amended, now "given N Projects", thanks. 
\\


\textbf{\em
1.21 What is P in Fig 3?  There are some outer loops missing.  Give the complete algorithm.}\\

Perhaps it was referring to the P in Fig 4?
Pi refers to the performance of predictor-i and Pj refers to the performance of predictor-j, basically it counts the number of wins/ties/losses. 

\textbf{\em 
1.22 illusive should be elusive} \\

Amended, thanks. \\

\textbf{\em
1.23 p3 "It is no longer necessary to work on simulated data (as done by Shepperd and Kadoda [32]".  The reason simulated data were used was so that specific characteristics could be built in along with a known true model.} \\

Thank you, we have modified the sentence. \\

\textbf{\em
1.24 Enlarge Fig 6.  It's important.} \\

Amended, thanks. \\

\textbf{\em
1.25 'jitter'????? p14} \\

Amended, thanks. \\

\textbf{\em
1.26 p15
depreciated should be deprecated}\\

Amended, thanks. 

