\clearpage
\section*{Replies to reviewer comments}

Thank you for your detailed reviews on the previous version. They were most helpful. 
Replies to comments are below.

\subsection*{Editor}

{\em The empirical study produces a negative result
- which is generally quite welcome. However, the reviewers (and I)
question the generality of your conclusions! Does your result mean
that ``Kernel density estimation'' must not be investigated any further, or...
}

We completely agree with your concerns- the empirical basis of the prior draft
was too restrictive for us.
In this version, we repeated our study on 19 data sets (not the 3 we saw before) and the
same result holds.

Also, we were somewhat too adamant in the previous draft.
We made our suggestion/conclusion more clear in this version.
The last sentence of Abstract reads:
\begin{quote}
``Hence, -provided that similar experimental settings are adopted- we discourage the use
of kernel methods as a weighting strategy in ABE.
 ''
\end{quote}
Also the last sentences of the $1^{st}$ paragraph in Section \ref{sec:conclusions} states:
\begin{quote}
``We hesitate to discourage further research with different experimental settings or with different datasets.
However, if similar use of kernels is to be adopted, we do not recommend the use of kernel methods as a weighting strategy in ABE.''
\end{quote}
This makes it clear that we are only talking about ABE for effort estimation
and not (as might be read in the
previous draft), kernel methods for all SE applications.\\

{\em 
If you think that you can accommodate these concerns (especially
the concern of Reviewer 2), please submit a major revision".}

We highly value your invitation for a re-submit after a major revision.
Therefore, we paid particular attention to address every comment of reviewers.
After following through the invaluable comments of reviewers, this study has become much more sound and to the point.
Furthermore, we improved the study with further experiments: Previous version used $3$ datasets evaluated with $2$ performance measures, whereas this version makes use of $19$ datasets evaluated with $3$ error measures; previous version used $330$ ABE variants, whereas this version reports $2090$ variants.
We believe that it is now ready for a re-review.
Below is a list of our replies to every individual comment.


\subsection*{Reviewer 1:}
{\em
The article describes an analysis of a specific type of analogy-based estimation,
i.e., kernel-density estimation for non-uniform weights of analogies. The article is
very well written and structured. It is easy to read.

}

Thank you very much, we are very pleased with your comments. \\

{\em 

Although the article focuses on an important topic,
I have several concerns:

}

We fully agree with all your concerns. 
We went through all your comments one by one and restructured our work accordingly.
Following is a list of our replies. \\

{\em 
The motivation why kernel-density estimation was chosen as object of study
is quite weak and not convincing to me as motivation for performing a comprehensive study. 
The more obvious reason that the technique could be specifically suited for software engineering
projects and data is not addressed.
}

The previous paper was, as you say, very weak on motivation. To repair that, we have
added new motivation section (Section \ref{sec:motivation}) that explains why we performed this study.\\ 

{\em 
The statement ``ABE methods are never enhanced by a particular kernel method" seems to be based on some kind of proof. However, it could also stem from your observations. I recommend to describe in more detail what the (potentially empirical basis) for this statement is.
}

You are right, we should have made that statement more clear. Our claim depends on empirical evaluation of $2090$ ABE variants (it was $330$ in the previous draft) according to various performance criteria such as MdMRE, Pred(25) and win-tie-loss values. Furthermore, in this version we included MAR criterion in accordance with the feedback we received from another reviewer. We explain this claim very early, now the last two sentence of Abstract reads as: 
\begin{quote}
After an extensive experimentation of $19$ datasets, $3$ evaluation criteria, $5$ kernels, $5$ bandwidth values and a total of $2090$ ABE variants, we found that: 1) non-uniform weighting through kernel methods cannot outperform uniform weighting ABE and 2) kernel type and bandwidth parameters do not produce a definite effect on estimation performance.
Hence, -provided that similar experimental settings are adopted- we discourage the use of kernel methods as a weighting strategy in ABE.\\
\end{quote}


\textit{ Section 2.1 I recommend to also integrate hybrid methods (i.e., methods based on
data *and* expert judgment such as CoBRA. I also recommend to include some studies that show that the accuracy can be significantly improved by maintaining the estimation model over time. It would be nice for the reader if such kind of concept is also part of your work.}


Thank you very much for reminding us an important background work. 
We happily included CoBRA like integrated approaches and their iterative applications.
Now the last sentences of the first paragraph in Section \ref{subsec:software-effort-estimation} is:

\begin{quote}
There are also successful applications where expert and model based techniques are
integrated to complement one another [7,21]. In particular when such estimation practices are employed iteratively over time, the estimation accuracy can significantly be improved.
For example in [40], an integrated approach called CoBRA was applied in an iterative
manner and estimation accuracy was improved from 120\% error down to 14\%.\\
\end{quote}


{\em The so-called ``experiments" are mainly data analyses of existing repositories. This seems to be reasonable to a certain extend. However, it is quite a weak evaluation due to a lack of knowledge about context and assumptions. I recommend for future work to do a customization
for a specific company and to derive real problems afterwards.
}

We agree that analysis in a company setting where we would tackle with a real problem would be a very interesting future direction. Therefore, we included it as the third element of our Future Work Section. 
Now the third future direction in the paper is:

\begin{quote}
 3. Customization: Experimentation in this paper is based on publicly available datasets.
Our knowledge about the context and assumptions of these datasets are limited to
their documentation. One further direction to this research could be customizing
this work for a particular company and derive the actual problems associated with
non-uniform weighting schemes.\\
\end{quote}

{\em 
Overall, the scope (technique, purposes, domain) is very limited and probably only of minor interest for most of the EMSE journal readers.
}

We agree with the reviewer that this paper will be of interest only to those working in the effort estimation area.

However, those researchers do publish in EMSE.  Jorgensen and
Shepperd~\cite{Jorgensen2007}
report that of the  76 journals  that published effort estimation
results, this journal is ranked in the top four (they list a dozen
EMSE publications related to effort estimation).  Also, this query
``\url{http://www.springerlink.com/content/1382-3256/?k=cost+estimation&o=10}''
yields dozens of papers relating to cost/effort estimation in this
journal.

\subsection*{Reviewer 2:}
{\em First of all I would like to applaud the authors' candor about the negative results rather than trying to dress them up in some way.
}

Thank you very much for your positive feedback. \\

{\em 
Nevertheless, I do have some serious problems.
}

Thanks to your and other reviewers' comments, we re-structured our paper and enriched it with further experiments: Previous draft was reporting $330$ ABE variants evaluated on $3$ datasets, whereas this version of the paper reports $2090$ ABE variants evaluated on $19$ datasets. Below is the list of your comments and our replies to them.\\

{\em 
First, though we need to deal with a potential publication bias against negative results, the work still needs to be interesting.  
}

That was also the concern of other reviewers and we are thankful for that comment, we should have stated our motivation more explicitly. 
In the new version of our paper, we included a whole section about our motivation (Section \ref{sec:motivation}) after the Introduction. \\

{\em 
There's an excellent discussion of these kind of issues in [1]. 
}

We are grateful for that reference. We went through all \textit{5} short papers in that reference and positioned our paper in the context of negative results according to our summary of these papers.
We devoted a whole section (Section \ref{sec:negative-results}) concerning this issue and its relation to our paper.
Section \ref{sec:negative-results} reads:

\begin{quote}
While it would have been gratifying
to have found a positive result (e.g. that some kernel method was very much better),
it is important to report such negative results as well.
A very thorough discussion on the value of negative results can be found in \cite{Browman1999}.
The fundamental question is whether a negative result poses a positive knowledge. Positive knowledge is defined by Browman et al. to be the ability of being certain, not being either right or wrong~ \cite{Browman1999}.
However, not all certain conclusions are \textit{knowledge} per se.
Common concerns are: 
\bi
\item[\textbf{\textit{i.}}] is the topic/hypothesis plausible, 
\item[\textbf{\textit{ii.}}] are the experiments sound, 
\item[\textbf{\textit{iii.}}] do the results propose \textit{``negative evidence''} or \textit{``non-conclusive search''} and 
\item[\textbf{\textit{iv.}}] will the reported results be valuable to future research.
\ei
As for \textbf{\textit{i.}}, research on weighting methods in ABE is quite plausible, see weighting method proposed earlier by Mendes et al.~\cite{Mendes2002, Mendes2003}.
In that respect, our evidence of negative results serve the purpose of guiding research away from conclusions (such as kernel weighting can improve ABE performance) that would otherwise seem reasonable~\cite{Browman1999}.

When presenting negative evidence it is crucially important to have \textit{sound and extensive} experimentation (condition \textbf{\textit{ii.}}).
This report rigorously investigates kernel weighting on \textit{19} datasets subject to \textit{3} performance measures through appropriate statistical tests.

The idea behind condition \textbf{\textit{iii.}} is that \textit{``one should disvalue inconclusive results''}~\cite{Browman1999}, i.e. negative conclusions are more meaningful than \textit{uncertainty}. 
The kernel weighting experiments of this paper on a wide range of ABE variants are negative evidence to conclude that it does not improve ABE performance, thereby satisfying \textit{\textbf{iii.}}
Finally condition \textit{\textbf{iv.}} questions the benefit of results to future research.
After years of research, effort estimation still suffers from conclusion instability.
For stable conclusions, retiring a considerable portion of search space is as important as the discovery of successful applications.
The contribution of this work is through retirement of \textit{2090} of ABE variants.
\end{quote}

{\em 
The situation we have is:
``Another possibility is that a novel hypothesis occurs to a scientist. This scientist proceeds to test. his bright idea only to discover that it is mistaken. Might he or she publish these findings to ward off other scientists who might be tempted to pursue this dead end?"
The answer revolves around how interesting the results to which I'd answer not that interesting.  My suggestion is a short note of perhaps 20\% of the length of this paper.
}

Thanks to your valuable feedback. As an aside, we do mention that we have been talking to Briand and Basili about a
{\em fast abstracts} section of ESE containing very small papers that need not have (e.g.) extensive background notes. To date,
they have not been responsive so we are stuck with the current norms in scientific publishing; i.e. as much as possible,
each paper has to be a stand-alone unit.

Nevertheless, we have significantly pruned this paper, from 29 pages to 21. And that is with
 new experiments (MAR performance measure is included) and much more data
(the dataset number is increased from $3$ to $19$) and corrections.\\

{\em 

The second problem is the paper isn't very well written - it reads like an early draft as it's often note like or lacks articles etc etc (below I've identified a few typos but there are a lot more). 

}

We are sorry for that, we corrected all the below and did our best to present you a much better draft this time. \\

{\em
 More seriously it's just hard to follow and a lot of irrelevant information is included eg in the intro why discuss feature selection when the paper is about analogy weighting and selection. 
}

We removed that claim together with other less-relevant explanations. \\

{\em  
Nor I do get any sense of why analogy weighting is an important and hard problem.  
}

We wrote a new section (Section \ref{sec:motivation}) so as to explain the motivation behind tackling the problem of analogy weighting. \\

{\em  
 Section 2 is weak eg why have 2 different taxonomies of related work -- one will do!  
}

We completely agree, we integrated these 2 sub-sections into 1 and included new references. \\

{\em  
 The discussion and Eqn 2 (for inverse rank weighting) seems unnecessary -- it's pretty obvious.
 }

You are right, we removed the discussion as well as the equation. \\

{\em 
Your discussion implies k $>$1 since k=1 $\rightarrow$ U-ABE equiv N-ABE
}

That is very true, thanks for reminding us to explicitly state that. The $4^{th}$ sentence of the second paragraph in Section \ref{sect:expr} states:
\begin{quote}
Note that $k>1$ for $\forall k$, because for $k=1$ U-ABE and N-ABE would be equivalent.\\
\end{quote}

{\em 
p12 Much as I appreciate the work of Kitchenham and her co-workers this new definition of dataset quality seems desperately over-simplistic.
}

We dropped that claim. \\

{\em 
MMRE? Given the authors are aware of its limitations why not do the statistical tests on absolute residuals?  Why pred(25)?
}


Quite correct- we included absolute residual experiments in this version.
However, we elected to {\em add} the one you suggest rather than {\em delete} the ones you depreciate since
(as the reviewer might appreciate), other commenters on this paper might complain ``why did they not use
standard measures in wide use in the literature''? As for Pred(25) we made a small note in Section 5.4:

\begin{quote}
Another alternative performance measure is PRED(25), which is reported to be one of the most commonly used accuracy statistics~\cite{Kitchenham2001}. \\
\end{quote}

{\em 
Minor points:
p4,  l36: problem explicitly addressed by Delphi where individual estimates are kept  anonymous.
}

We removed that sentence.\\

{\em 
p6, l22: a a
}

Corrected.\\

{\em 
p6, final sentence: meaning is unclear.
}

We corrected this sentence as follows:

\begin{quote}
We want to remind researchers, who want to follow afore mentioned future directions that those future directions may as well end up in negative results. \\
\end{quote}

{\em 
p7, l27: statistical algorithms?
}

It is removed from the paper. \\

{\em 
p9, l45: started
}

It is removed from the paper. \\

{\em 
p17, l42: what were the imputed values (so the work could be replicated?)
}

The \textit{``best-k''} approach is a pseudo-best k and it changes for different test instances, i.e. since we select ten random test instances and choose the lowest error yielding \textit{k} value as the so called \textit{``best''}, the values are somewhat random. We are repeating experiments 20 times for each setting and we are reporting that we do 20 runs to remove possible bias issues.
Therefore, we reported our methodology instead of fixed \textit{best-k} values, which might change for the next set of runs when another researcher is replicating our work. \\

{\em 
p24, l37: does hardly let??
}

It is removed from the paper. \\

{\em 
p26, l46: the choice of performance measure is not an open issue. See Kitchenham et al. [2].  
The only issue is what characteristic of accuracy you are actually interested in, normally spread of residuals (error-proneness) and central tendency (bias) would be sufficient.
}

We dropped that claim. \\

{\em 
REFERENCES:
[1]     H. Browman, "Negative results," Marine Ecology Progress Series, vol. 191, pp. 301-309, 1999.
[2]     B. A. Kitchenham, S. G. MacDonell, L. Pickard, and M. J. Shepperd, "What accuracy statistics really measure," IEE Proceedings - Software Engineering, vol. 148, pp. 81-85, 2001.
}

Thank you very much for these references, they were most helpful.
We included them into the paper.
