

\documentclass{svjour3}                     % onecolumn (standard format)
%\documentclass[smallextended]{svjour3}     % onecolumn (second format)
%\documentclass[twocolumn]{svjour3}         % twocolumn
%
\smartqed  % flush right qed marks, e.g. at end of proof
%
\usepackage{graphicx}
\usepackage{rotating}
\usepackage{color}
\usepackage{verbatim}
\usepackage{fancyvrb}
\usepackage{algorithmic}
\usepackage{url}
\usepackage{cite}
\usepackage{multirow}
\usepackage{rotating}
\usepackage{subfigure}
\usepackage{float}
\restylefloat{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newcommand{\bi}{\begin{itemize}}
\newcommand{\ei}{\end{itemize}}
\newcommand{\be}{\begin{enumerate}}
\newcommand{\ee}{\end{enumerate}}
\newcommand{\tion}[1]{\S\ref{tion:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{itemize}
   \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{itemize}}
\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}

\begin{document}

\title{Kernel Methods for Software Effort Estimation%\thanks{Grants or other notes
%about the article that should go on the front page should be
%placed here. General acknowledgments should be placed at the end of the article.}
}
\subtitle{Effects of different kernel functions and bandwidths on estimation accuracy}

%\titlerunning{Short form of title}        % if too long for running head

%\author{Ekrem Kocaguneli        \and
%        Tim Menzies \and
%        Jacky W. Keung
%}
%
%\authorrunning{Short form of author list} % if too long for running head
%
%\institute{E. Kocaguneli and T. Menzies\at
%              Lane Department of Computer Science and Electrical Engineering \\
%              West Virginia University\\
%              Morgantown, WV 26505, USA\\
%              \email{ekocagun@mix.wvu.edu, tim@menzies.us}
%           \and
%           Jacky W. Keung \at
%			Department of Computing\\
%              The Hong Kong Polytechnic University\\
%              Kowloon, Hong Kong\\
%              \email{jacky.keung@comp.polyu.edu.hk}  
%}

\date{Received: date / Accepted: date}
% The correct dates will be entered by the editor


\maketitle

\begin{abstract}
Analogy based estimation (ABE) generates an effort estimate for a new  software project
through adaptation of similar past projects (a.k.a. analogies).
Majority of ABE methods follow uniform weighting in adaptation procedure.
In this research we investigated non-uniform weighting through kernel density estimation.
%We used different kernel methods to compute non-uniform weights of analogies.  
After an extensive experimentation of $19$ datasets, $3$ evaluation criteria, $5$ kernels, $5$ bandwidth values and a total of $2090$ ABE variants, we found that: 1) non-uniform weighting through kernel methods cannot outperform uniform weighting ABE and 2) kernel type and bandwidth parameters do not produce a definite effect on estimation performance.
Hence, -provided that similar experimental settings are adopted- we discourage the use of kernel methods as a weighting strategy in ABE.
 

\keywords{Effort estimation, data mining, kernel function, bandwidth}
\end{abstract}

\section{Introduction}
\label{sec:introduction}

%Software effort estimates are reported to be often wrong by a factor 
%of four~\cite{Boehm1981} or even more~\cite{kemerer87}. 
%%The critical results of wrong estimates for a company are obvious: 1) promising projects that would stay within budget may be rejected, 2) accepted projects may over-run their planned resources and worst of all 3) over-running projects may be cancelled thereby wasting the entire effort.
%Wrong estimates can have critical results for a company such as cancellation of budget over-running projects, thereby wasting the entire effort.
%Therefore, effort estimation is an active research area~\cite{Boehm2000,Jorgensen2007,Shepperd2007,Mendes2007,Li2009} that constantly explores more variations.  
%

If a researcher or an industrial practitioner reads the literature on software effort estimation,
they will encounter a dauntingly large number of different estimation methods.
For example, the following is a partial list of some of the methods currently being used:
\bi
\item Boehm uses linear regression~\cite{Boehm1981}.
\item Shepperd prefers
analogy-based methods~\cite{Shepperd1996}. 
\item
Auer et al.~\cite{Auer2006} uses extensive search to find
weights for project features. 
\item
Pendharkar et al. used Bayesian Network (BN) for effort estimation and incorporated BN into decision making procedure against risks~\cite{Pendharkar2005}.
\item
Mendes and Mosley used a data-driven Bayes net for web effort estimation~\cite{Mendes2008a}.
\item
Li et al. combine feature weighting with  instance selection~\cite{Li2009}.
\ei
This list is hardly  complete. Elsewhere~\cite{Menzies2010xx,koc2011}, 
we have studied all the different kinds
of
{\em Analogy-Based Estimation} (hereafter, ABE) methods in the literature:
\bi
\item 
ABE generates estimates by sampling the neighborhood of
some test instance. 
\item The exact sampling method is controlled by the
{\em kernel method} which we divide into {\em uniform} and {\em non-uniform}
(here after U-ABE and N-ABE, respectively).  
\item
Uniform methods treat all
items in the local neighborhood in the same way.
\item
Non-uniform methods weight
those neighbors in different ways.
\ei
Our list of ABE variants is presented in the next section:
we can find over 17,000
different ways to configure ABE-style estimators.
How we select the right method from this large menagerie of possibilities?
One way is to try many options, then see what works best on the local data.
Baker then Menzies et al.,
used exhaustive search through all project features, learners and other variables ~\cite{baker07,Menzies2006}.
The CPU intensive nature of that approach begs the question: is there a simpler way?

There is indirect evidence that there must be a simpler way.
If we look at the size of the training data available for effort estimation,
it is usually only a few dozen (or less) instances. For example:
\bi
\item
The data accessible to researchers in four recent publications~\cite{Mendes2003,Auer2006,baker07,Li2009} have
median size of $13,15,33,52$, respectively. 
\item
Later in this paper we list the 19 data sets used in this study: they vary in size from 10 to 93
with a median of 28.
\ei
Given that the size of these data sets is so small, it seems
unreasonable to believe that (e.g.) complex multi-dimensional
partitioning schemes can find more than more simplistic methods
such as ``take the median of the variables in the local region''.  

This is indeed the case.
The experiments of this paper show that, for ABE,
the simplest kernel schemes are {\em most often as good as anything 
else}. 
This is an important result since it
means that future
researchers have less to explore (at least, in the field of ABE).
Hopefully, if more researchers critically reviewed the space of options for their tools,
then
we will arrive at a much smaller and much more manageable set of candidate
effort estimation methods.

The rest of the paper is organized as follows.
To focus the paper, we will answer the following research questions:
\bi
\item[RQ1] Is there evidence that non-uniform weighting improves the performance of ABE?
\item[RQ2] What is the effect of different kernels for non-uniform weighting in ABE?
\item[RQ3] What is the effect of different bandwidths?
\item[RQ4] How do the characteristics of software effort datasets influence the performance of kernel weighting in N-ABE?
\ei
To do so,
Section \ref{sec:motivation} summarizes our motivation of this story.
Section \ref{sec:negative-results} talks about the value of negative results.
In Section \ref{sec:background} we provide background information regarding related work.
Section \ref{sec:methodology} explains the adopted experimental methodology.
In Section \ref{sec:results} results are presented.
Section \ref{sec:discussion} is a discussion section and in Section \ref{sec:threats-to-validity} the threats to the validity of the results are presented.
In Section \ref{sec:conclusions} we summarize our conclusions and answer the research questions.
Finally in Section \ref{sec:future-work} we list future directions of this research.
 
Note that,
for reasons of space,
our results will be presented in a summary format.
For the full results, see http://goo.gl/qpQiD.

\section{Motivation}
\label{sec:motivation}
This work is part of an on-going investigation into effort estimation.
Effort estimation is important since, 
those estimates
are often wrong by a factor of four~\cite{Boehm1981} or even more~\cite{kemerer87}. As a result, the allocated funds may be inadequate to develop the required project. In the worst case, over-running projects are canceled and the entire development effort is wasted. 

We study analogy-based estimation (ABE), for several reasons:
\bi
\item It
is a widely studied approach~\cite{Auer2006,Walkerden1999,%
Kirsopp2003,Shepperd1997,Shepperd1996,Kadoda2000,Li2008,Li2006,Li2007,Li2009,
keung2008a,keung2008b,keung2008c}. 
\item 
It works even if the domain data is sparse~\cite{Myrtveit}.
\item 
Unlike other predictors, it makes no
assumptions about data distributions or an underlying model.
\item
When the local data does not support
standard algorithmic/parametric
models like COCOMO, ABE can still be applied.
\ei
Based on a literature review of just one sub-section of
the field~\cite{Menzies2010xx}, we have found at least eight dimensions
that distinguish different ABE methods:
\bi
\item[$A$]: The distance  measure used to compute similarity;
\item[$B$]: The ``neighborhood'' function that decides what is a ``near'' neighbor;
\item[$C$]: The method used to summarize the nearest neighbors;
\item[$D$]: The instance selection mechanism;
\item[$E$]: The feature weighting mechanism;
\item[$F$]: The method for handling numerics, e.g. logging, discretization, etc.
\ei
That review found in the literature
 three to nine variants of $A,B,C,D,E,F$
(which combine to a total over 17,000
variants).  Some of these variants can be ruled out, straight
away. For example, for ABE that reasons only about the single nearest
neighbor, then all the summarization mechanisms return the same
result.  Also, not all the feature weighting techniques require
discretization, thereby further decreasing the space of options.
However, even after discarding some combinations, there are still
thousands of possibilities to explore. 

In our view, it is unacceptable that researchers continually extend effort
estimation methods without
trying to prune away the less useful variants. To that end,
in previous work, we have tried to rank
and prune estimation methods based
on model selection~\cite{Menzies2010xx} or
feature
weighting~\cite{Menzies2006} or instance selection~\cite{koc2011}.

We have had much recent success in pruning different variants:
\bi
\item
For
COCOMO-style data~\cite{Boehm1981},
only four variants were
demonstrably better than the another 154 variants~\cite{Menzies2010xx}.
\item
Also, in non-COCOMO data, we have found 13 variants that perform better
than 77 others~\cite{koc2011jj}.
\ei
This paper is our first exploration
of kernel methods. Kernel methods are important since they comment on many
of the options within $A,B,C,D,E,F$ listed above:
\bi
\item
Simpler
kernel methods mean simple neighborhood and summarization methods.
\item
In theory, better estimates could be generated by a smarter sampling of the neighborhood.
For example, an intelligent selection of the kernel might compensate for data scarcity. 
\ei
We study kernel estimation since, 
if the effort data corresponds to a particular distribution, then it would seem wise to bias that
sampling by that distribution.
Also,
at least one other research team
in the field of effort estimation have also begun exploring
different kinds of kernel methods~\cite{Mendes2002,Mendes2003}.

Despite the potential of kernel  methods to improve effort estimation,
it is an unexplored area.
For the most part,
researchers in this area propose kernel methods
with  minimal motivation or experimentation~\cite{Briand1999,jeffery2001,Mendes2002,Mendes2003}.
Hence, prior to performing the experiments of this paper, we believed that kernel
methods would be a rich source of future insights into effort estimation:
\bi
\item
The space of sampling and weighting schemes seen in the SE literature is much smaller than that seen 
in other fields (see for example the  literature from data mining or signal processing~\cite{Palpanas2003, John1995, Frank03}).  
\item
Hence,
it seemed to us that 
a rigorous exploration of this under-explored area might be a worthy topic of research, perhaps applying methods
not yet used in the SE literature.
\ei
This paper presents that rigorous exploration and leads to the
the negative result summarized in the conclusion (that simple kernel methods do as
well as  anything else, at least for ABE). 

\section{On the Value of Negative Results}
\label{sec:negative-results}

While it would have been gratifying
to have found a positive result (e.g. that some kernel method was very much better),
it is important to report such negative results as well.
A very thorough discussion on the value of negative results can be found in \cite{Browman1999}.
The fundamental question is whether a negative result poses a positive knowledge. Positive knowledge is defined by Browman et al. to be the ability of being certain, not being either right or wrong~ \cite{Browman1999}.
However, not all certain conclusions are \textit{knowledge} per se.
Common concerns are: 
\bi
\item[\textbf{\textit{i.}}] is the topic/hypothesis plausible, 
\item[\textbf{\textit{ii.}}] are the experiments sound, 
\item[\textbf{\textit{iii.}}] do the results propose \textit{``negative evidence''} or \textit{``non-conclusive search''} and 
\item[\textbf{\textit{iv.}}] will the reported results be valuable to future research.
\ei
As for \textbf{\textit{i.}}, research on weighting methods in ABE is quite plausible, see weighting method proposed earlier by Mendes et al.~\cite{Mendes2002, Mendes2003}.
In that respect, our evidence of negative results serve the purpose of guiding research away from conclusions (such as kernel weighting can improve ABE performance) that would otherwise seem reasonable~\cite{Browman1999}.

When presenting negative evidence it is crucially important to have \textit{sound and extensive} experimentation (condition \textbf{\textit{ii.}}).
This report rigorously investigates kernel weighting on \textit{19} datasets subject to \textit{3} performance measures through appropriate statistical tests.

The idea behind condition \textbf{\textit{iii.}} is that \textit{``one should disvalue inconclusive results''}~\cite{Browman1999}, i.e. negative conclusions are more meaningful than \textit{uncertainty}. 
The kernel weighting experiments of this paper on a wide range of ABE variants are negative evidence to conclude that it does not improve ABE performance, thereby satisfying \textit{\textbf{iii.}}
Finally condition \textit{\textbf{iv.}} questions the benefit of results to future research.
After years of research, effort estimation still suffers from conclusion instability.
For stable conclusions, retiring a considerable portion of search space is as important as the discovery of successful applications.
The contribution of this work is through retirement of \textit{2090} of ABE variants.

\section{Background}
\label{sec:background}

%In this section, we will provide general background information about software effort estimation and ABE.
%We will also address how kernel methods have been utilized in the literature and discuss how they can be adapted to software effort estimation domain as a weighting strategy.

\subsection{Software Effort Estimation}
\label{subsec:software-effort-estimation}
We can divide software effort estimation into at least two groups~\cite{Shepperd2007}: {\em expert judgment} and {\em model-based} techniques.
{\em Expert judgment} methods depend on consolidation of expert opinions regarding the cost of a new project and are widely used in software effort estimation practices~\cite{Jorgensen2004}.
Expert judgment can be applied either explicitly (following a method like Delphi\cite{Boehm2000}) or implicitly (informal meetings among experts).
%Regardless of the method expert judgment is applied, it is prone to some pitfalls.
%One possible pitfall in expert-based methods is the fact that they are open to clashes of interest.
%For instance a faulty estimation of a senior expert may be taken over the more accurate estimation made by a junior expert.
%Another pitfall is that expert-based methods can be as good as your experts.
%Jorgensen et al. report that human experts are very poor at evaluating and improving their estimation skills~\cite{Jorgensen2009}. 
Unlike expert-based methods, {\em model-based techniques} do not rely heavily on human judgment.
Model based techniques are products of:
\bi
\item[1)] Algorithmic and parametric approaches or 
\item[2)] Induced prediction systems.
\ei
The first approach is the adaptation of an expert-proposed model to local data.
A widely known example to such an approach is Boehm's COCOMO method~\cite{Boehm1981}.
The second approach is particularly useful in the case where local data does not conform to the specifications of the expert's method.
A few examples of induced prediction systems are linear regression, neural nets, model trees and analogy based estimation~\cite{Menzies2006, Kadoda2001}.
There are also successful applications where expert and model based techniques are integrated to complement one another~\cite{Briand1997, Klas2008}.
In particular when such estimation practices are employed iteratively over time, the estimation accuracy can significantly be improved. 
For example in ~\cite{Trendowicz2006}, an integrated approach called CoBRA was applied in an iterative manner and accuracy was improved from $120\%$ error down to $14\%$. 
%Regardless of the categorization of models, they are all built on inherent assumptions.
%For example, linear regression assumes that the effort data fits a straight line while model trees assume that the data fits a set of straight lines.
%In the cases where data violates these assumptions, patches are applied, e.g. take the logarithm of exponential distributions before linear regression \cite{Boehm1981, Kitchenham2009}.
%However, choosing the appropriate patch again requires qualified experts.

Analogy based estimation (ABE) or estimation by analogy (EBA) is a form of case based reasoning (CBR) and it is grouped together with induced prediction systems.
%In their 2005 study Myrtveit et al. follow a different categorization than the one presented in this paper~\cite{Myrtveit}.
%They group estimation models into sparse-data and many-data categories.
%Sparse-data methods are defined to be estimation methods that need few or no historical data.
%Examples to sparse-data methods are Analytical Hierarcy Process (AHP)~\cite{Shepperd2001}, expert judgment and case-based reasoning.
%Many-data methods are identified in the form of a function and are subdivided into: 1) functions, 2) Arbitrary function approximators (AFA).
%The functions may be in the form of $y=Ax^B$, where a mathematical relationship exists between the variables of the expression (e.g. linear regression models).
%Unlike functions, AFA make no assumption between predictor and response variables.
%EBA, classification and regression trees (CART) and artificial neural networks (ANN) belong to this class~\cite{Myrtveit}.
%
% 
%According to the taxonomy presented by Myrtveit et al. CBR may belong to both sparse-data or many data category ~\cite{Myrtveit}.
%If one uses CBR to reason from and already selected case then it is identified to be a single-data method.
%However, if CBR is used to identify the closest case, then it is categorized as a many-data method.
%ABE is an example of this use of CBR~\cite{Myrtveit}.
ABE generates its estimate for a new project by gathering evidence from similar past projects.
When we analyze the previous research of experts on the domain of ABE such as Shepperd et al.~\cite{Shepperd1996}, Mendes et al.~\cite{Mendes2003} and Li et al.~\cite{Li2009}, we can see a baseline technique lying under all ABE methodologies. The baseline technique is composed of the following steps:
\bi
\item Form a table (training set) whose rows are completed past projects and whose columns are \textit{independent} variables (the features that define projects) and a \textit{dependent} 
variable (the recorded effort value). 
\item Decide on the number of similar projects (\textit{analogies}) to use from the training set, i.e \textit{k}-value.
\item For each test instance, select \textit{k} analogies out of the training set.
\bi
\item While selecting analogies, use a similarity measure (for example the Euclidean distance).
\item Before calculating similarity, apply a scaling measure on independent features to equalize their influence on this similarity measure.
\item Use a feature weighting scheme to reduce the effect of less informative features.
\ei
\item Adapt the effort values of the \textit{k} nearest analogies to come up with an effort estimate.
\ei
Following the steps of this baseline technique, we define a framework called ABE0.
ABE0 uses the Euclidean distance as a similarity measure, whose formula is given in Equation \ref{equ:euclid}.
In Equation \ref{equ:euclid}, $w_i$ corresponds to feature weights applied to independent features.
ABE0 framework does not favor any features over the others, therefore each feature has equal importance in ABE0, i.e. $w_i=1$.

\begin{equation}\label{equ:euclid}
Distance = \sqrt{\sum_{i=1}^{n}{w_i(x_{i} - y_{i})^{2}}}
\end{equation}

The next step is deciding on how to adapt project costs.
There is a wide variety of adaptation strategies in the literature~\cite{Mendes2002a}.
Using effort value of the nearest neighbor~\cite{Briand1999}, taking mean~\cite{Myrtveit1999} or median~\cite{Angelis2000} of closest analogies, inverse distance and inverse rank weighted mean of closest analogies are among the commonly used adaptation methods~\cite{Mendes2002a}. 
%The adaptation of effort suggested by baseline approach does not have to be a a complex process.
Angelis et al. suggest that as the number of the closest projects increase, median is a robust solution~\cite{Angelis2000}.
They have found that taking median instead of mean decreases the estimation error. 
We want the estimates of ABE0 framework to represent the majority of selected instances and not greatly affected by extreme values.
Therefore, ABE0 returns the median effort values of the \textit{k} nearest analogies.
Since ABE0 implicitly assigns equal weights to \textit{k} nearest analogies, it turns out to be an U-ABE method.
%in our research is due to the fact that we also make use of high as well as low \textit{k} values and using mean could have let extreme effort values have a strong influence on the estimation.
%However, we want the estimates of ABE0 framework to represent the majority of selected instances and not greatly affected by extreme values.
%Therefore, ABE0 uses median instead of mean.

In this research we will compare the results of ABE0 framework with different non-uniform weighting strategies, i.e. with different N-ABE methods.
Note that since ABE0 is a framework for U-ABE methods, in the rest of the paper the two terms will be used interchangeably.
%The word \textit{weighted} in WABE may at first be considered to refer to both \textit{weighting attributes} as well as \textit{weighting analogies}.
%However, ABE0 framework already includes a mechanism for weighting independent attributes (see Equation \ref{equ:euclid}).
%Therefore, when we talk about WABE, weighting will refer to weighting of instances rather than features.
N-ABE methods have been previously addressed in literature.
For example inverse rank weighted mean (IRWM) was proposed by Mendes et al. ~\cite{Mendes2003}.
IRWM method enables higher ranked analogies to have greater influence than the lower ones.
Assuming that we have \textit{3} analogies, the closest analogy (CA) gets a weight of $\frac{3}{\sum_{i=1}^{3}{i}}$, the second closest (SC) gets a weight of $\frac{2}{\sum_{i=1}^{3}{i}}$ and the last analogy (LA) gets $\frac{1}{\sum_{i=1}^{3}{i}}$.
%With this weighting approach, IRWM would calculate the estimation as in Equaiton \ref{equ:irwm}.
%Note that we can generalize IRWM to handle more than $3$ neighbors as follows: In the case of $n$ closest analogies, the closest neighbor would have the weight of $n$, the next one would have the weight of $n-1$ and so on.
%The weighted sum would then be divided by the sum of all weights: $\sum_{i=1}^{n}{i}$
%\begin{equation}\label{equ:irwm}
%Effort = (3*CA + 2*SA + 1*LA) / (3 + 2 + 1) 
%\end{equation}
%Being inspired by N-ABE methods like IRWM, in this research we question whether it is possible to develop an automated N-ABE approach.

\subsection{Kernel Density Estimation}
\label{subsec:kernel-estimation}

%IRWM is one example of a broad class of statistical reasoning called kernel density estimation, where IRWM acts like a triangular kernel assigning weights to analogies on the basis of their distance.
%However, IRWM-like methods have their roots in expert judgment.
%In the lack of valuable experts, such a weighting strategy would be almost impossible to apply to the needs of a particular dataset.
%Kernel density estimation on the other hand is a non-parametric estimation method that is used to uncover the underlying structures of data, which a parametric approach may fail to reveal~\cite{Wand1994}.
%Since we used the univariate kernel density estimation, we will suffice to mention the univariate case in this paper. 
%However, the same approach can be easily adapted to higher dimensionalities~\cite{Wand1994, Scott1992}.

%Assuming that we are given a sample $X_1,...,X_n$ with a continuous, univariate density $f$, the kernel density estimator is defined as in Equation \ref{equ:kde}.

%\begin{equation}\label{equ:kde}
%\hat{f}(x,h) = \frac{1}{nh} \sum_{i=1}^{n}{K\left(\frac{x-X_i}{h}\right)}  
%\end{equation}
%
%In Equation \ref{equ:kde}, $K$ is defined as the \textit{kernel} and $h$ is defined to be the bandwidth.

The kernel function is usually chosen to be unimodal and symmetric about zero~\cite{Wand1994}, such as a probability distribution function.
In a kernel estimation method, the center of the kernel is placed right on each data point and the influence of data points is distributed to the overall neighborhood.
To reach the final density function, contributions coming from all data points are summed up.
For example, note how individual Gaussian curves add up to generate the final density estimate in Figure \ref{fig:pointwise-kernel} (one kernel is added for each observation along the x-axis~\cite{Scheid2004}).

\begin{figure}
\begin{center}
\includegraphics[width=0.8\textwidth]{pointwise-kernel.png}
\end{center}
\caption{We see a Gaussian kernel density estimate built on individual data points. Each point is in the center of a kernel and its effect is distributed to its neighborhood. The sum of all kernels make up the final Gaussian kernel density estimate.}\label{fig:pointwise-kernel}
\end{figure}

Kernel density estimation has been successfully used for different type of datasets.
For instance Palpanas et al. use kernel density estimation to address the problem of deviation detection in environment of sensor networks~\cite{Palpanas2003}. 
Frank et al. use kernel estimation for locally weighting the attributes of Naive Bayes~\cite{Frank03}.
%thereby relaxing the independence assumption~\cite{Frank03}.
Furthermore John et al. use kernel estimation to tackle the normality assumption regarding continuous datasets~\cite{John1995}. 
%They replace single Gaussian distribution that is used to model continuous data with non-parametric kernel density estimation and they report considerable improvements on real and artificial datasets.
%Although kernel density estimation is used in different areas for modeling different types of data, to the best of our knowledge it was not previously used in the context of ABE.
%In this research we propose using kernel density estimation as a N-ABE method for assigning weights to selected analogies.

%For WABE, apart from the \textit{k} value we have different parameters that can be tuned: kernel type and bandwidth.
%Previously it is reported that the choice of kernel does not have a significant effect on the performance~\cite{Cressie1993}.
%However, this statement is valid for spatial data and the effect of different kernels have not been investigated for software effort data.
%Therefore, in our research we inlcluded different types of kernels to observe the effect of kernel selection on effort data.

The kernels we use in our research are: Uniform, triangular, Epanechnikov and Gaussian.
We can use a generic formula for some kernels, which is given in Equation \ref{equ:general-kernel}, where $\textbf{1}_{(|x| < 1)}$ is the indicator function.
Furthermore, Equation \ref{equ:general-kernel-b} and Equation \ref{equ:general-kernel-g} explain for the calculation of other functions in Equation \ref{equ:general-kernel}.
Depending on the value of $p$ in Equation \ref{equ:general-kernel}, we can derive different kernels.
For example for $p=0$ we elicit the uniform kernel, for $p=1$ we elicit Epanechnikov kernel etc.

\begin{equation}\label{equ:general-kernel}
K(x,p) = \frac{\left( 1-x^2 \right)^p}{2^{2p+1}B\left( p+1, p+1 \right)} \textbf{1}_{(|x| < 1)}
\end{equation}

\begin{equation}\label{equ:general-kernel-b}
B\left( p+1, p+1 \right) = \frac{\Gamma\left(a\right)\Gamma\left(b\right)}{\Gamma\left(a+b\right)}
\end{equation}

\begin{equation}\label{equ:general-kernel-g}
\Gamma\left( n\right) = \left( n-1\right)!
\end{equation}

\begin{figure}[!b]
\centering
{
\renewcommand{\arraystretch}{2.5}
\begin{tabular}{l | c } 
\renewcommand{\baselinestretch}{0.8}
\textbf{Kernel Type} & \textbf{Formula} \\\hline 
Uniform Kernel & $K(\rho) = \frac{1}{2}\textbf{1}_{(|\rho| < 1)}$  \\\hline
Triangular Kernel  & $K(\rho) = \left( 1 - |\rho| \right) \textbf{1}_{(|\rho| < 1)}$  \\\hline
Epanechnikov Kernel  & $K(\rho)=\frac{3}{4} \left( 1 - \rho^2 \right) \textbf{1}_{(|\rho| < 1)}$ \\\hline
Gaussian Kernel  & $K(\rho)=\frac{1}{\sqrt{2\pi}}e^{(\frac{-1}{~2}\rho^2)}$ \\
IRWM Kernel  & ---\\
\end{tabular}}
\caption{The formulas for different kernels used in this study. In formulas $\rho = \frac{x - X_i}{h}$. Note that IRWM kernel has different characteristics and its calculation details were provided in Section \ref{subsec:software-effort-estimation}.}
\label{fig:kernel-formulas}
\end{figure}

This paper explores the kernels
of
Figure \ref{fig:kernel-formulas}
as well as 
IRWM~\cite{Mendes2002, Mendes2003}.
%The general shapes of these kernels are given in Figure \ref{fig:kernels}~\cite{Scheid2004}.
IRWM is not actually proposed as a kernel method and it does not fully conform to the definition of standard kernel methods.
%(being unimodal and symmetric about zero and so on).
However, due to the weighting strategy it proposes we can read it as an expert proposed kernel.

%\begin{figure}[ht!]
%  \centering
%  \subfigure[Uniform Kernel]{\label{fig:uniform-kernel}\includegraphics[width=50mm, height=30mm]{uniform-kernel.png}}                
%  \subfigure[Triangular Kernel]{\label{fig:triangular-kernel}\includegraphics[width=0.32\textwidth]{triangular-kernel.png}}
%  \subfigure[Epanechnikov Kernel]{\label{fig:epanechnikov-kernel}\includegraphics[width=0.32\textwidth]{epanechnikov-kernel.png}}  \\
%  \subfigure[Gaussian Kernel]{\label{fig:gaussian-kernel}\includegraphics[width=0.3\textwidth]{gaussian-kernel.png}}
%   \subfigure[IRWM Kernel for \textit{k}=10]{\label{fig:irwm-kernel}\includegraphics[width=0.3\textwidth]{irwm-kernel.png}}
%  \caption{General shapes of the kernel types used in this research. Note that IRWM is not symmetric unlike other kernels, that is due to the fact IRWM has a different weighting procedure than other kernels (see Equation \ref{equ:irwm}).}
%  \label{fig:kernels}
%\end{figure}


%Initially, we planned to explore many more kernels than those listed above.
%However, to our surprise, we found that
%estimation accuration was not influenced by the kernel method. 
%A literature review revealed that a similar effect has been reported in domains other than  effort estimation.
A literature review revealed that
the selection of bandwidth (\textit{h}) for kernels is more influential than the  kernel types~\cite{Scott1992, Cressie1993}.
Bandwidth \textit{h} is fundamentally a scaling factor that controls how wide probability density function will spread, i.e. appropriate choice of \textit{h} is critical to avoid under and over-smoothing~\cite{Scheid2004, Wand1994}.
To avoid both under and over-smoothing conditions we used various bandwidth values.
One of the bandwidths we used is suggested by John et al., which is $h=1/\sqrt{n}$ where $h$ is the bandwidth and $n$ is the size of dataset~\cite{John1995}.
The other bandwidth values we used are: 2, 4, 8 and 16.

%\begin{figure}[h!t]
%  \centering
%  \subfigure[h=0.1]{\label{fig:bandwidth-1}\includegraphics[width=0.45\textwidth]{bandwidth1.png}}                
%  \subfigure[h=0.05]{\label{fig:bandwidth-05}\includegraphics[width=0.45\textwidth]{bandwidth05.png}}  
%  \subfigure[h=0.02]{\label{fig:bandwidth-02}\includegraphics[width=0.45\textwidth]{bandwidth02.png}}
%  \subfigure[h=0.005]{\label{fig:bandwidth-005}\includegraphics[width=0.45\textwidth]{bandwidth005.png}}
%  \caption{We see the effect of bandwidth on kernel density estimation. From Figure \ref{fig:bandwidth-1} to Figure \ref{fig:bandwidth-005}, the bandwidth gets smaller and smaller and we observe a transition from over-smoothing to under-smoothing.}
%  \label{fig:choice-of-bandwidth}
%\end{figure}

%We can see how choosing different bandwidth values affect kernel density estimation in Figure \ref{fig:choice-of-bandwidth}.
%Figure \ref{fig:choice-of-bandwidth} from \cite{Scheid2004} is suggestive, but not convincing evidence that effort estimation may not be improved by the selection
%of the kernel method.
%Prior work (reporting that bandwidth \textit{h} values
%were more important than choice of kernel) comes from (e.g.) the
%signal processing literature~\cite{Scott1992,Cressie1993}.  Data
%sets in those domains are very different to those in
%effort estimation: thousands to billions of instances (for signal
%processing) versus dozens to hundreds of instances (for effort
%estimation). 
%In data-started
%domains like effort estimation, it may be the case that an intelligent selection of the
% kernel could compensate for data scarcity. 
% Note that other authors
% in the field of effort estimation have also shared this
% opinion~\cite{Mendes2002,Mendes2003}.
%However, we are unaware of any effort estimation publications that explore this issue in a systematic manner.
%Hence, the rest of this paper. 
%
%\ssection{Motivation}
%
%Software effort estimation is a very big research area and we have still discovered only a small portion of it.
%With every new method proposed, we discover \textit{dos and don'ts} of it.
%We adopt the beneficial methods and retire the others, thereby consistently discovering the estimation domain.
%However, the tendency is usually to report the attempts that yield improvements on current methods and let go of the promising trials that did not result in considerable improvements.
%In other words mostly we hear only the \textit{``dos''} of the domain.
%\textit{``Don'ts''} part usually go unreported.
%However, it is fundamentally important to report the results of all promising approaches, regardless of whether it outperforms previous methods or not.
%Because they may save time in another similar study or may inspire new directions.
%
%In this research we elaborate on weighting in ABE and propose a novel method for weighting analogies.
%For weighting we use kernel density estimation.
%Kernel density estimation has the advantage of being a non-parametric estimation method that can uncover particular properties of a dataset.
%It is reported to yield significant improvements in various settings~\cite{Palpanas2003, Frank03, Frank03}, but its effects are not addressed in effort estimation domain.
%Software effort datasets also have particular challenges due to their inherent characteristics and we need to figure out different ways to discover these characteristics.
%We conduct extensive experiments with various kernels and try multiple bandwidths for each kernel in our research.
%At the end we cover a considerable number of settings, which were never adressed or reported before in effort estimation domain.
%Although kernel methods have yielded relatively successful results in different domains, we did not observe a significant improvement for ABE.
%Basing on the fact that other researchers may or will be conducting similar studies, we think that our results can give hints regarding kernel methods for weighting in ABE.
%%Furthermore, in this research we question the reasons why kernel methods do not have similar characteristics for software effort data.
%
%\clearpage
\section{Methodology}
\label{sec:methodology}

%In this section we provide the methodology that we adopted in our research.
%We discuss how we use kernel density estimation as a non-uniform weighting method as well as which kernels we use for this purpose.
%Furthermore, we provide information regarding the datasets we used in this research and discuss their characteristics.
%Also we provide information regarding the experimental settings we adopted.
%Finally we discuss the performance criteria according to which we compare the performance of N-ABE methods to basic U-ABE method ABE0.


\subsection{Weighting Method}
\label{subsec:weighting-method}

%Here we summarize how kernel density estimation is employed as a non-uniform weighting method.
Assume that our dataset is divided into two sets: $A = \left\lbrace x_1,..., x_k \right\rbrace$ (selected \textit{\textbf{A}}nologies) and $R = \left\lbrace t_1,...,t_{n-k} \right\rbrace$ (\textit{\textbf{R}}est of the dataset).
We build the kernel density estimation on $R$ and evaluate the resulting function at instances of $A$.
Equation \ref{equ:wkde} shows the probability calculation with kernel density estimation.
In Equation \ref{equ:wkde} the kernel $K$ is built on training data $t_i \in R$ and is evaluated at $k^{th}$ analogy ($x_k$) for a bandwidth of $h$.
%The general idea of this approach is that selected \textit{k} analogies for a test instance come from a distribution and this distribution is specific to the dataset.
%Furthermore, according to this specific distribution we get different probability values for each analogy.
%In other words, we have different $f(x_k,h)$ values for each analogy. 
After scaling these probability values to 0-1 interval according to Equation \ref{equ:weight}, we use them as weights for analogies.
%, where $x_k$ represents all the analogies in $A$ except $x_i$.
After calculating $weight_{x_i}$  for each analogy, we update their actual effort values according to Equation \ref{equ:effort-update}. 
%Updating the actual effort values simply means multiplying an effort value with its weight.
%Equation \ref{equ:effort-update} shows how actual effort value of an analogy, $x_i$, is updated.
%In our research WABE approaches use the updated effort values for adaptation to estimate the effort of a test instance.

\begin{equation}\label{equ:wkde}
f(x_k,h) = \frac{1}{nh} \sum_{t_i \in R}{K\left(\frac{x_k-t_i}{h}\right)}  
\end{equation}

\begin{equation}\label{equ:weight}
weight_{x_i} = \frac{f(x_i,h) - max(f(x_k,h))}{max(f(x_k,h)) - min(f(x_k,h))}
\end{equation} 

\begin{equation}\label{equ:effort-update}
updatedEffort_{x_i} = actualEffort_{x_i} * weight_{x_i} 
\end{equation} 


\subsubsection{Uniform vs. Non-Uniform Weighting}

%In this research we compare two different weighting strategies in ABE: Uniform weighting and non-uniform weighting.
%For the ease of naming we use abbreviations to refer to these strategies.
%\textbf{U}niform weighting ABE is abbreviated by U-ABE, whereas \textbf{N}on-unform weighting ABE is abbreviated by N-ABE.
%
%For both strategies we can come up with a number of different methods.
%As for U-ABE, we defined a base method that we call ABE0 (see Section \ref{subsec:ABE}).
%ABE0 assumes that each one of the \textit{k} analogies have equal importance: It assigns equal weights to all analogies.
%Therefore, ABE0 uses the actual effort values without applying any change during adaptation for the final estimate.
%In terms of N-ABE methods, we use different kernel types in the context of kernel density estimation.
%Depending on the underlying characteristics of a particular dataset, kernel density estimation assigns certain probability values to analogies.
%These probability values define the importance of each analogy: Higher probability means higher weight.
%Different kernel types used as N-ABE methods in our research are: Uniform kernel, triangular kernel, Epanechnikov kernel, Gaussian kernel and IRWM.

The fundamental difference between N-ABE and U-ABE methods is that in U-ABE analogies are given uniform weights and their actual effort values are used in an \textit{as is} manner, whereas in N-ABE analogies are assigned different weights and their actual effort values are multiplied by these weight values.
As for U-ABE, we defined a base method that we call ABE0 and for N-ABE we use $5$ different kernel methods.

One point that needs further clarification is the use of uniform kernel as a N-ABE method.
%Since uniform kernel also follows a uniform weighting scheme, it may be confused with ABE0. 
%However, the uniform weighting scheme of ABE0 and that of uniform kernel are quite different from one another.
%In uniform kernel, depending on how we select bandwidth some of the instances get uniform and non-zero probability values, whereas some others get a probability value of zero.
%Therefore, uniform kernel in fact turns out to be a N-ABE method.
Figure \ref{fig:weight-vs-non-weight} succinctly illustrates the difference between uniform kernel being a N-ABE method and ABE0 being a U-ABE method.
%In Figure \ref{fig:weight-vs-non-weight} x-axis shows the instance IDs, whereas y-axis shows probabilities of instances simulated by a normal probability distribution function (PDF).
ABE0 assumes equal importance of \textit{all} instances and assigns equal probabilities. 
A uniform kernel would assign equal non-zero probabilities to \textit{only a certain portion} of the instances, whereas the rest of the instances would be assigned a weight of zero (i.e. they would be ignored).

\begin{figure}[th!]
  \centering
  \label{fig:weighting-abe}\includegraphics[width=0.8\textwidth]{plots/uniform-vs-non-uniform-weighting.png}
  \caption{In the case of ABE0 all instances are given equal probability values, hence equal weights. However, uniform kernel prefers some instances over the others: Only a certain portion of the instances are given equal non-zero weights.}
  \label{fig:weight-vs-non-weight}
\end{figure}



\subsection{Data}

In our research, we have used $19$ datasets, most of which are heavily used in software effort estimation research: Nasa93, the original 
Cocomo81 \cite{Boehm1981}, Desharnais\cite{Desharnais1989} and so on.
%Cocomo81 and Nasa93 datasets contain projects developed in NASA, whereas Desharnais dataset contains projects developed by Canadian software houses.
%A number of studies using these datasets subject them to pre-processing steps such as feature and instance selection.
%Since observing the impact of instance and/or feature selection is not one of our aims, in our research we did not resort to these pre-processing steps and used all features as well as all the instances in the datasets.
%Furthermore, studies making use of Desharnais dataset exclude 4 of the project instances due to the fact that some of the project properties are missing.
%Yet, the datasets are already sparsely populated and there are ways to handle missing entries.
Note that $4$ projects in Desharnais dataset has missing entries, we used imputation~\cite{Alpaydin2004} to handle them.
%Simple mean imputation is one of these methods, in which missing properties are replaced with the mean values of other projects for the same property~\cite{Alpaydin2004}.
%To handle instances with missing entries in Desharnais dataset, we used simple mean imputation.
%Apart from selecting commonly used datasets, we took the quality of the datasets into consideration.
%In order to evaluate the goodness of datasets, Kitchenham and Mendes propose a quality scoring that consists of four values: poor (less than ten projects), 
%fair (between ten to twenty projects), good (between twenty to forty projects) and excellent (more than forty projects) \cite{Kitchenham2007}.
%Following this quality criteria all the datasets we use in our research rank as excellent quality.
The details regarding these datasets can be found in Figure \ref{fig:datasets}.

\begin{figure}
\centering
{\scriptsize 
\begin{tabular}{l|rrl|l}  
 \textbf{} & \textbf{} & \textbf{} & \textbf{} & \textbf{}\\
 \textbf{Dataset} & \begin{sideways}\textbf{Features}\end{sideways} & \begin{sideways}\textbf{Size}\end{sideways} & \textbf{Description}&\textbf{Units}\\ \hline
  Cocomo81 &17&63& NASA projects& months\\
  \hspace{4 mm}Cocomo81e &17&28& Cocomo81 embedded projects&months\\ 
  \hspace{4 mm}Cocomo81o &17&24& Cocomo81 organic projects&months\\ 
  \hspace{4 mm}Cocomo81s &17&11& Cocomo81 semi-detached projects&months \\ 
  Nasa93 &17&93& NASA projects&months\\ 
  \hspace{4 mm}Nasa93\_center\_1 &17 &12 & Nasa93 projects from center 1&months \\ 
  \hspace{4 mm}Nasa93\_center\_2 &17&37& Nasa93 projects from center 2&months \\ 
  \hspace{4 mm}Nasa93\_center\_5
   &17&40& Nasa93 projects from center 5&months \\ 
  Desharnais &12&81& Canadian software projects&hours \\ 
  \hspace{4 mm}DesharnaisL1 &11 &46 & Desharnais projects  using Language1 & hours\\ 
  \hspace{4 mm}DesharnaisL2 &11 &25 & Desharnais projects using Language2 & hours\\ 
  \hspace{4 mm}DesharnaisL3 &11 &10 & Desharnais projects using  Language3 & hours\\ 
  SDR &22&24& Turkish software projects& months\\ 
  Albrecht &7&24& Projects from IBM&months\\ 
  Finnish &8 &38 &Finland projects &hours  \\
  Kemerer &7 &15 &Large business applications  &months  \\
  Maxwell &27 &62 &Projects from  commercial banks in Finland  &hours \\
  Miyazaki94 &8 &48 &Japanese COBOL projects &months \\
  Telecom &3 &18 &Maintenance projects for telecom companies  &months  \\\cline{2-3}
 & Total& 699
\end{tabular}}
\caption{We used 699 projects coming from 19 datasets.
Datasets have different characteristics in terms of the number of attributes as well as the measures of these attributes.}
\label{fig:datasets}
\end{figure}


\subsection{Experiments}
\label{sect:expr}

Our experimental settings aim at comparing the performance of standard U-ABE (ABE0) to that of N-ABE.
% (non-uniform weighting through different kernel types).
To separate train and test sets we use leave-one-out method, which entails selecting $1$ instance out of a dataset of size $n$ as the test set and using the remaining $n-1$ instances as the training set.
For each test instance, we run ABE0 and N-ABE separately and store their estimates.
%Then we run different N-ABE methods for the same test instance and store their estimates as well.
As the analogy number is reported to play a critical role in estimation accuracy~\cite{Kadoda2000}, both for U-ABE and N-ABE methods, we tried different \textit{k} values.
Furthermore, to hinder any particular bias that would come from the settings of a single experiment, we repeated the afore mentioned procedure 20 times.

We use 2 forms of ABE methods (uniform and non-uniform weighting) induced on $19$ datasets for 5 different \textit{k} values.
The \textit{k} values we used in our research are: $k \in \left\lbrace 3,5,7,9,best \right\rbrace$.
$best$ is a  pseudo-best \textit{k} value that is selected for each individual test instance through a process, in which we randomly pick up $10$ instances from the training set and select the lowest error yielding \textit{k} value as the $best$.
Note that $k>1$ for $\forall k$, because for $k=1$ U-ABE and N-ABE would be equivalent.
In addition, we use 5 different kernels (Uniform, Triangular, Epanechnikov, Gaussian and IRWM) with 5 bandwidth values in N-ABE experiments.
To further explore field of software effort estimation, we investigate a total of $2090$ different settings in this research:
\bi
\item U-ABE Experiments: 95 settings
\bi
\item \textit{19 datasets * 5 k values = 95}
\ei
\item N-ABE Experiments: 1995 settings
\bi
\item Standard Kernels: \textit{19 datasets * 5 k values * 4 kernels * 5 bandwidths = 1900}
\item IRWM: \textit{19 datasets * 5 k values = 95}
\ei
\ei

\subsection{Performance Criteria}
Performance measures comment on the success of a prediction.
For example, mean absolute residual (MAR) is the mean of absolute residuals (the difference between the predicted and the actual):
{\begin{equation}\label{ar}
MAR = mean(AR_i={|actual_i-predicted_i}|)
\end{equation}}

%To observe the effect of non-uniform weighting in ABE, we use the following performance measures: the magnitude of relative error (MRE), median magnitude of relative error (MdMRE), prediction within 25\% error rate (Pred(25)), absolute residual (AR) and win-tie-loss values generated by a statistical test (Mann-Whitney U Test).
Magnitude of relative error (MRE) is one of the most commonly used performance criterion for assessing the performance of competing software effort estimation methods~\cite{Molokken-Ostvold2004, Briand1999, Foss}.
%Furthermore, as we can see from Formula \ref{EquationMRE}, MRE value is a direct measure of the absolute difference between the prediction and actual value~\cite{Stensrud} and hence it gives a per-instance based performance evaluation.

\begin{equation} 
MRE = \frac{|actual_{i} - predicted_{i}|}{actual_{i}}
\label{EquationMRE} 
\end{equation}

Median MRE (MdMRE) has emerged as the de facto standard evaluation criterion for cost estimation models~\cite{Stensrud2002}.
%Often software effort studies make use of mean magnitude of relative error (MMRE) as the performance criterion as well.
%MMRE is the mean of all MRE values.
%However, the mean approach considers every observation and is sensitive to individual predictions that have high MREs~\cite{Mendes2003}.
%One way to address this problem is the median approach via MdMRE.
Median also gives information about central tendency and is less sensitive to extreme MRE values.
%Therefore, while we comment on the results of MRE-based measures in Section \ref{subsec:mre-based-results}, we provide MdMRE values.
MdMRE formula is given in Equation \ref{EquationMdRE}, where $n$ is the test set size.

\begin{equation} 
MdMRE = median(MRE_1, MRE_2,...,MRE_n)
\label{EquationMdRE} 
\end{equation}

%Pred(\textit{l}) is also a very widely used performance criterion in software effort estimation studies~\cite{Mendes2003}.
%Pred(\textit{l}) is defined as the prediction at level \textit{l}, which measures
%the percentage of estimates that are within l percent of the actual effort values~\cite{Mendes2008a}.
%The suggestion regarding the value of \textit{l} is that it shall be set at 25 percent~\cite{Conte1986} and that a good prediction system should offer this accuracy level 75 percent of the time~\cite{Conte1986, Mendes2003}.
%Therefore, we will also use Pred(\textit{l}) at 25 25\% (i.e. Pred(25)).

Another alternative performance measure is PRED(25), which is reported to be one of the most commonly used accuracy statistics~\cite{Kitchenham2001}.
It is defined as the percentage of predictions falling within 25\% of the actual values~\cite{Mendes2008a}:
{\begin{equation}\small
PRED(25) = \frac{100}{N}\sum_{i=1}^N\left\{\begin{array}{ll}
\mbox{1 if $MRE_i \le \frac{25}{100}$}\\
\mbox{0 otherwise}
\end{array}\right.
\end{equation} 
}
For example, PRED(25)=50\% implies that half of the estimates are failing within 25\% of the actual values \cite{Shepperd1997}.

\begin{figure}[!h]
\renewcommand{\baselinestretch}{1}
\begin{algorithmic} \footnotesize
\IF{~Mann-Whitney(\textit{$P_i$}, \textit{$P_j$}, 95) says they are the same}
	\STATE \textit{$tie_{i}$} = \textit{$tie_{i}$} + 1;
	\STATE \textit{$tie_{j}$} = \textit{$tie_{j}$} + 1;
\ELSE
   	\IF{~better( median(\textit{$P_{i}$}), median(\textit{$P_{j}$}))}
   		\STATE \textit{$win_{i}$} = \textit{$win_{i}$} + 1
   		\STATE \textit{$loss_{j}$} = \textit{$loss_{j}$} + 1
   	\ELSE
   		\STATE \textit{$win_{j}$} = \textit{$win_{j}$} + 1
   		\STATE \textit{$loss_{i}$} = \textit{$loss_{i}$} + 1 \ENDIF \ENDIF
\end{algorithmic}
\caption{Comparing algorithms (\textit{i},\textit{j})
on performance ($P_i$,$P_j$). The ``better'' predicate changes according to $P$.
For error measures like MRE, ``better'' means lower medians. However, for PRED(25),
``better'' means higher medians.}
\label{FigureWinTieLossPseudocode}
\end{figure}


If performance measures are used as a stand-alone evaluation criteria (i.e. not combined with appropriate statistical tests), results may lead to biased or even false conclusions~\cite{Foss}.
Therefore, we use the so called $win,tie,loss$ statistics, whose pseudocode is given in Figure \ref{FigureWinTieLossPseudocode}.
In Figure \ref{FigureWinTieLossPseudocode}, we first check if two
distributions $i, j$ are statistically different (Mann-Whitney rank-sum test, $95\%$ confidence);  otherwise we increment $tie_i$ and $tie_j$. 
If the distributions are statistically different, we update $win_i,win_j$
and $loss_i,loss_j$ after comparing the performance measures so as to see which one is better.



%
%Note that, MRE related measures are subject to many pitfalls.
%If MRE is used a stand-alone performance evaluation criterion (i.e. not combined with appropriate statistical tests), it may lead to biased or even false conclusions~\cite{Foss}.
%To prevent us from falling into MRE-related pitfalls, we use another performance criterion called win-tie-loss calculation.
%A win-tie-loss calculation tells that comparison between two methods $i$ and $j$ makes sense only if they are statistically different.
%If there is no statistically significant difference between two methods, say method $i$ and method $j$, then it indicates that results are observations coming from the same distribution.
%Therefore, methods are said to `\textit{`tie''} and their $tie$ values ($tie_i$ and $tie_j$) are incremented.
%However, if there is a statistical difference between two methods, then the method with a lower median MRE score, say $i$, is said to have a ``win" and the one with the lower MRE, say $j$, is said to have a ``lose". 
%The related values $win_i$ and $loss_j$ are incremented by one.
%The pseudocode for a win-tie-loss calculation is given in Figure~\ref{FigureWinTieLossPseudocode}.
%For the comparison of methods in win-tie-loss calculation, a non-parametric statistical test (the Mann-Whitney rank-sum test) is used at a significance level of 95\%.

\section{Results}
\label{sec:results}

To see the effect of kernel weighting, we studied 19 datasets and 3 different performance measures.
For each performance measure we tried 4 different kernels subject to 5 different bandwidths, plus the IRWM kernel (which does not have a bandwidth concept) and reported associated $win,tie,loss$ statistics.

\input{sampleTable}

\fig{win-tie-loss-desharnais} shows  a sample of our results. It  reports the $win,tie,loss$ 
statistics of Desharnais dataset for ABE0 and N-ABE through Gaussian kernel. 
For each dataset we have $4$ such tables (one for each kernel), so for all datasets there are \textit{19 Datasets $\times$ 4 tables = 76 tables}.
For the IRWM kernel there will be another \textit{19 datasets $\times$ 1 kernel = 19 tables}.
Hence,  in total, our results comprise \textit{76 + 19 = 95 tables} to report. All these tables are available on line
at 
 http://goo.gl/qpQiD.
However, for space reasons, we summarize those tables as follows.

\input{allDatasetResults}

In \fig{win-tie-loss-desharnais} we see that each row reports $win,tie,loss$ statistics of ABE0 methods (\textit{k=3,5,7,9,best}) as well as N-ABE methods (\textit{k=[3,5,7,9,best]+kern} where 
\textit{kern} stands for kernel weighting) subject to a particular performance measure.
Similarly, each column shows the $win,tie,loss$ statistics associated with a particular bandwidth value.
As can be seen in \fig{win-tie-loss-desharnais}, for Desharnais dataset ABE0 methods always have higher $win$ values and always have a $loss$ value of $0$, meaning that they never lose against N-ABE methods.
That is, by summarizing each row/column intersection we can see that the performance of ABE0 is never improved by N-ABE.

\fig{summary-all1} and \fig{summary-all2} repeat that summarization process for all 19
datasets and all kernel/bandwidth combination:
\bi
\item
Each row of these summary
figures shows the comparison of ABE0 performance to that of N-ABE subject to 3 different performance measures.
\item
Every kernel/bandwidth intersection in \fig{summary-all1} and \fig{summary-all2} has 3 symbols corresponding to MdMRE, MAR and Pred(25) comparisons from left to right.
\item
Each of these 3 symbols can have 3 values: $-,+,o$.
\begin{itemize}
\item
\textit{``$-$''} means that N-ABE decreased the accuracy of ABE0;
\item 
\textit{``$o$''} means ABE0 and N-ABE are statistically same;
\item
\textit{``$+$''} shows that ABE0 accuracy was improved through kernel weighting (i.e. N-ABE has a better performance than ABE0).
\ei
\ei
%To test if ABE0's performance is improved ($+$), decreased
%($-$) or not changed ($o$) by N-ABE we see, =m===look at all $95$
%tables such \fig{win-tie-loss-desharnais} and see the behavior
%of 5 \textit{k} values under ABE0 and N-ABE.  
We assign the symbols ``$+$'' or ``$-$'' if the performance associated
with the majority of the k-values (at least 3 out of 5)
are improved or degraded by N-ABE (in
terms of $win-loss$). If there is no change, 
we assign a ``$o$'' symbol to that setting.

Observe that in all these summaries:
\bi
\item 
There is only one dataset (SDR in \fig{summary-all2}) where N-ABE provides a performance improvement
in certain cases.  Even for that dataset there are $15$ ``$+$''
symbols and $21$ ``$-$'' symbols, meaning that most of the time
N-ABE is still destructive.  
\item
In $18$ other datasets, which is $\frac{18}{19}
= 95\%$ of all the datasets, there is not a single case where N-ABE
improves the performance of ABE0. 
\ei
Note that these summary tables 
contain results from different performance criteria (MdMRE,
MAR, Pred(25)) as well as kernels and bandwidths.  Therefore, our
conclusion from \fig{summary-all1} and \fig{summary-all2} is that that \textit{``non-uniform weighting
through standard kernel methods does not improve the performance
of ABE''} holds in the majority case across different datasets and error
measures.



\input{irwm-results}

Another summary table is given in \fig{summary-irwm}.
\fig{summary-irwm} is very similar to \fig{summary-all1} in the sense that it summarizes the performance of N-ABE over 19 datasets w.r.t. three different performance measures.
The difference is that \fig{summary-all1} summarizes the results of standard kernel methods, whereas in \fig{summary-irwm} we see the N-ABE performance under an \textit{expert-based} kernel, i.e. IRWM.
Although there are important differences between standard and expert-based kernels (IRWM has no bandwidth parameter), the results seen in \fig{summary-irwm} is quite similar to those of \fig{summary-all1}.
As can be seen in \fig{summary-irwm}, there is not a single case where N-ABE (under IRWM kernel) improves the performance of ABE0.
Furthermore, the amount of ``$-$'' symbols is much more than ``$o$'', meaning that N-ABE decreases the performance of ABE0 most of the time.

\section{Discussion}
\label{sec:discussion}

%Rigorous experimentation presented in this research shows that our results are mostly negative, i.e. different variants of kernel estimation do not increase estimation accuracy.
Our pre-experimental intuition was that non-uniform weighting could end up performing no better and perhaps even worse than simple uniform weighting for effort estimation datasets.
A simple example for such a negative intuition is provided in Figure \ref{fig:intuitive-example}.

\begin{figure}[th!]
\centering
{ 
\begin{tabular}{l|c|c|c} 
\textbf{Analogies}   & $P_1$ & $P_2$ & $P_3$ \\\hline
\textbf{Effort Values} & 10.0 & 20.0 & 60.0 \\\hline
\textbf{Probabilities} & 0.1 & 0.3 & 0.5 \\\hline
\textbf{Weights} & 0.0 & 0.5 & 1.0 \\\hline\hline
\textbf{Uniform Weighting Estimate} & \multicolumn{3}{c}{20.0}\\\hline
\textbf{Non-Uniform Weighting Estimate} & \multicolumn{3}{c}{35.0}\\
\end{tabular}}
\caption{An intuitive example. In a 3 analogy case, there is a $75\%$ change for a hypothetical test project between uniform and non-uniform weighting.}
\label{fig:intuitive-example}
\end{figure}

Assume that for a test project in Figure \ref{fig:intuitive-example}, 3 analogies are chosen ($P_1$, $P_2$, $P_3$) with effort values of $10.0$, $20.0$ and $60.0$.
Also assume that a kernel assigned the probabilities of $0.1$, $0.3$ and $0.5$ to these analogies respectively.
When these probability values are normalized, the weights assigned to $P_1$, $P_2$, $P_3$ become $0.0$, $0.5$ and $1.0$. 
The estimation (i.e. median of analogies) for uniform weighting case would be $20.0$, whereas the estimation for non-uniform weighting case would be $35.0$.
A shift from an estimate of $20.0$ to $35.0$ is a dramatic change of $75\%$ rather than a slight correction.
Therefore, our intuition was that non-uniform weighting in software effort estimation could be disruptive rather than constructive.

Then the most likely question to be raised is ``Why do other fields~\cite{Palpanas2003, Frank03, John1995} benefit from weighting, whereas effort estimation does not?''.
Our belief is that the answer is partially hidden behind the low sample sizes of effort datasets.
Scarcity of the samples means that the weighting observes a signal being broadcast from a very small number of points in the neighborhood.
%Although we can tune the size of the neighborhood with the bandwidth value, in low sample sizes kernel estimation still performs poorly.
%
%We can observe the effect of instance size and bandwidth value on the estimation accuracy in Figure \ref{fig:sample-size-kernel-estimation}.
In Figure \ref{fig:sample-size-kernel-estimation} we simulate 50, 100 and 1000 samples coming from two Gaussian probability distribution functions (PDFs): $N(20,5)$ and $N(35,5)$.
Then we use kernel density estimation technique with a Gaussian kernel to estimate the density at discrete values of $x$ in [0-55] interval with a step-size of 1.
%In Figure \ref{fig:sample-size-kernel-estimation} the x-axis shows the instance IDs for discrete $x$ values and instance IDs are equal to actual $x$ values.
%An $x$ with the ID of say $10$ corresponds to the discrete $x$ value of 10.
%The y-axis in Figure \ref{fig:sample-size-kernel-estimation} shows the probability values for each $x$ instance.
%To better observe the probability values, we plot data as logarithmic scale for y-axis while still reporting the actual probability values rather than the logged values.
%The probability values of $x$ for the actual Normal distributions are shown with a solid line.
%Ideally we want to get estimates as close as possible to the envelope indicated by the solid line (actual PDFs). 

%The density estimates with different bandwidths (different $h$ values) are shown with different lines in Figure \ref{fig:sample-size-kernel-estimation}.
%%For example the dashed-line shows the estimate with a bandwidth of $0.001$ ($h=0.001$) and the dotted line shows the estimates for the bandwidth value of $10$ ($h=10$), whereas the solid line stands for the estimates made with the bandwidth value of $1$ ($h=1$).
%Three bandwidth values mean different neighborhood sizes and they enable different number of sample points to fall into neighborhoods.
%$h=0.001$ is an example of a too small bandwidth value.
%As we see, for all sample sizes $h=0.001$ does hardly let any points to fall into the neighborhood of $x$, therefore the probability is zero most of the time and whenever point(s) fall into neighborhood of $x$, we observe sparks of probability.
%Contrary to $0.001$, bandwidth value of $10$ is a too large value and it cannot model the underlying Gaussian distributions either.
%When we look at Figure \ref{fig:sample-size-kernel-estimation}, we see that estimations made with $h=10$ are over-smoothed and $h=10$ can only estimate a single Gaussian distribution instead of two.
%In this simple example $h=1$ yields the closest estimates to the actual probability values.
%However, we see that even for $h=1$, the sample size plays a critical role in estimation.
%When we look at the fit between $h=1$ estimates and the actual PDFs in Figure \ref{fig:sample-size-kernel-estimation}, we see that as we increase the size of simulated points the large deviations that we observe in Figure \ref{fig:50-sample-points} diminishes through Figure \ref{fig:1000-sample-points}.

In Figure \ref{fig:sample-size-kernel-estimation}, closest estimates require:
\bi
\item Optimum bandwidth (here $h=1$). Too small bandwidth ($h=0.001$) assigns most probability values (hence weights) to zero, whereas too big of a bandwidth ($h=10$) results in over-smoothing.
\item Considerable sample size. Note how optimum fit is achieved for a sample size of $1000$.
\ei


\begin{figure}[ht!]
  \centering
  \subfigure[50 Sample Points: Note the bad fit due to low sample size.]{\label{fig:50-sample-points}\includegraphics[width=0.88\textwidth]{plots/50samples2gaussians.png}}                
  \subfigure[100 Sample Points: Note the better fit due to increased sample size.]{\label{fig:100-sample-points}\includegraphics[width=0.88\textwidth]{plots/100samples2gaussians.png}}\\
  \subfigure[1000 Sample Points: Note the optimum fit due to high sample size.]{\label{fig:1000-sample-points}\includegraphics[width=0.88\textwidth]{plots/1000samples2gaussians.png}} 
  \caption{The effect of sample size and bandwidth on kernel density estimation. The choice of optimum bandwidth (\textit{h} value) is important. However, even with the optimum bandwidth, one still needs enough number of samples for successful estimation.
Sample size of $50$ appears to be too small and when we increase it to $100$, we get a better fit. Yet, for a very close fit, we need to go up to $1000$ sample points.}
  \label{fig:sample-size-kernel-estimation}
\end{figure}

In case of signal processing the sample sizes are closer to Figure \ref{fig:1000-sample-points} and as we see from the simulation example, kernel estimates can successfully model such densely populated datasets.
However, software effort datasets used in our research are similar to Figure \ref{fig:50-sample-points} and Figure \ref{fig:100-sample-points}.
%Hence Figure \ref{fig:50-sample-points} and Figure \ref{fig:100-sample-points} can give us hints about the kernel density estimation applied on software effort estimates.
When we observe behavior of kernel estimates for low sample sizes in those figures, it is somewhat expected to see lower performance values in sparsely populated data sets like software effort data sets.

%Although simulation study also confirms our intuitive feeling, data coming from simulation can hardly model the real-life data in software effort estimation domain and we wanted to observe whether results of intuitive examples would also hold for real-life datasets.
%Therefore, we conducted this research and rigorous experiments.



\section{Threats to Validity}
\label{sec:threats-to-validity}

We will address the threats to validity of this research under 3 categories: Internal validity, external validity and construct validity.

 \textit{Internal validity} asks to what extent the cause-effect relationship between dependent and independent variables holds~\cite{Alpaydin2004}. 
%The perfect case for the satisfaction of internal validity would be the application of a theory that was learned from past experiences to new situations.
%However, data in software effort estimation domain is a relatively sparse resource and most of the studies make use of commonly-explored datasets like the ones we use in this research.
%Therefore, the issue of internal validity thereatens all effort studies that use past data.
%However, we can mitigate this threat by simulating the behavior of a learned theory in new settings. 
We use leave-one-out method for all treatments to address internal validity issues. 
Leave-one-out selection enables us to separate the training and test sets completely in each experiment, thereby making the test sets completely new situations for the training sets.

\textit{External validity} questions the ability to generalize the results~\cite{Milic2004}.
To observe the generalizability of our results, we perform extensive experiments on 19 datasets.
The datasets are widely used in software effort estimation community and have very different characteristics in terms of various criteria such as size, number of features, types of features and measurement method.
%Furthermore datasets are subject to rigorous experimentation where we investigate the effects of non-uniform weighting on performance under $330$ settings.
%Our observations for all the settings are extremely similar.
%Therefore, for the datasets used in our research, our opinion is that the results have external validity.
However, to have full confidence in our claims, our study needs to be replicated by future studies.
%on other dataset and possibly with different non-uniform weighting strategies.

 \textit{Construct validity} (i.e. face validity) makes sure that we in fact measure what we intend to measure~\cite{Robson2002}. 
%The choice of performance measures is an open issue in software effort estimation domain.
Kitchenham et al. notes that different performance measures evaluate different aspects of prediction accuracy~\cite{Kitchenham2001}.
So as to assess N-ABE and ABE0 comparison from different standpoints, we made use of MAR, MdMRE and Pred(25) in our study.
However, as Kitchenham et al. points out in~\cite{Kitchenham2009}, it is wrong to solely use the performance measures without a statistical test.
%For example MdMRE is recognized as de facto evaluation criterion for cost estimation models~\cite{Stensrud2002} and it appears as a practical performance evaluation option to a number of researchers~\cite{Premraj2007, Lum2008, Li2006}.
%On the other hand, use MdMRE is still criticized for being unreliable~\cite{Myrtveit,Foss}.
%Foss et al. for instance shows that MRE can be misleding, if used as the only performance criterion~\cite{Foss}.
%Therefore, a study willing to have construct validity should not merely rely on MRE-based measures.
%To measure what we really intend to measure, we make use of win-tie-loss measures apart from MdMRE and Pred(25).
%Of course all these criteria have their inherent weaknesses and strenghts.
%Our aim in combining these measures is to use them in a complementary manner.
%For example strenght of MdMRE as well as Pred(25) is that they give a general picture of per instance-based evaluation.
%However, median error measure and Pred(25) become too general when averaged over 20 runs and we do not have much knowledge regarding individual runs.
Therefore, we also use win-tie-loss measures, where  we make use of Mann-Whitney U test at a significance level of 95\%.  
%
%on the other hand compare each method with one-another for each run and allow us to have a better opinion concerning individual runs.
%Furthermore, MdMRE and Pred(25) show us the difference between two methods based on MRE.
%But this difference may not always have statistical significance.
%To ensure the statistical validity of our results we make use of Mann-Whitney U test at a significance level of 95\% in win-tie-loss calculations.
%Therefore, our use of different performance measures such as MdMRE, Pred(25) as well as win-tie-loss, provides different perspectives of the results and lets us know if the results have statistical significance.

\section{Conclusions}
\label{sec:conclusions}

%The experiments shown above took three months to research, design, execute, then write up.
%It turns out that we could have spent the time more productively on other issues. 
%Our pre-experimental intuition that the right selection of kernel could compensate for scarce data in effort estimation data sets turned out to be incorrect. 
% XX i moved this text to future work section, where we wanted to say diplomatically that going after kernels would be futile


In this research we tried kernel density estimation as a non-uniform weighting strategy for ABE.
We conducted experiments with various kernels as well as bandwidths.
% the performance variations between U-ABE and N-ABE methods.
For the datasets and performance measures used in our research there was hardly any cases where N-ABE methods outperformed ABE0.
We hesitate to discourage further research with different experimental settings or with different datasets.
However, if similar use of kernels is to be adopted, we do not recommend the use of kernel methods as a weighting strategy in ABE.


Unlike studies in different domains that use kernel methods and report improved accuracy values~\cite{Palpanas2003, John1995}, we do not observe such an effect on software effort datasets.
The reason for different results may lie in different dataset characteristics.
For instance the datasets used in other domains are much more densely populated than software effort datasets.

%The literature is doubtful about the ability to make model rankings as offered in this paper between ABE0 and N-ABE~\cite{Foss, Myrtveit, Jorgensen2007}.
%For example Foss et al. comments that ``...is futile to search for the Holy Grail: A single, simple-to-use, universal goodness-of-fit kind of metric, which can be applied with ease to compare (different methods)''~\cite{Foss}.
%The evidence offered in literature that such rankings may change is based on:
%\bi
%\item the random selection of data,
%\item the dataset used,
%\item the evaluation metric used for comparison.
%\ei
%However, a convincing counter example to this scenario would be to show that method A is better than method B over multiple datasets, over multiple random selections of train/test data and over multiple evaluation criteria.
%In this research:
%\bi
%\item we used 3 of the most commonly utilized public datasets, 
%\item we adopted leave-one-out method in which every instance is considered as a test instance once,
%\item covered $330$ different settings,
%\item evaluated our results w.r.t. to 3 different evaluation criteria that have different inherent assumptions.
%\ei

%in terms of estimation accuracy it is possible to rank ABE0 better than kernel weighted N-ABE methods in software effort estimation.
%We know that software effort estimaton as well as ABE is a rigorously studied field and a lot of effort is invested in discovering the unknown space in this field.
%In this research we questioned a previously proposed idea of weighting instances in ABE methods.
%While weighting we used both the previously proposed method of IRWM as well as a novel method called kernel density estimation.
%Our research investigates a total of $330$ settings in this space, among which $315$ settings (settings of kernel weighting) were never investigated before or were not published.
%Our readings of our results is that weighting is not a very promising track to follow in ABE domain.
%Of course our resutls can be refuted with further studies.
%However, we wanted to share our findings and comments with the rest of the community so that they could have a hint regarding weighting in ABE domain and decide whether or not to invest too much effort in it.


\subsection{Answers To Research Questions}

%In this section we map the evaluation of our results to particular research questions that guided us in this research.

\textit{RQ1. Is there evidence that non-uniform weighting improves the performance of ABE?}
The results of our experiments do not show such an evidence.
On the contrary, for all settings ABE0 yields much better results than N-ABE methods. 
%Therefore, the evidence suggests that non-uniform weighting decreases estimation accuracy in ABE systems.

\textit{RQ2. What is the effect of different kernels for non-uniform weighting in ABE?}
%Similar studies on different data types report that change of kernels does not have a significant effect on the results~\cite{Cressie1993, Palpanas2003}.
%We observe inconsistent and extremely limited effect due to change of kernels.
There are only slight variations in performance when different kernels are used.
However, these variations do not follow a definite pattern and they are far from being considerable.
%Therefore, we can not say that different kernels have a definite effect on the performance of N-ABE methods.
%We can attribute the slight variations of performance between different kernels to different train and test combinations in different runs.

\textit{RQ3. What is the effect of different bandwidths?}
%Unlike the limited effect of different kernel types, choice of different bandwidths are reported to have a significant effect on the estimation performance~\cite{Cressie1993, Palpanas2003}.
%However, for software effort estimation domain -at least on the datasets used in this research- we did not observe such an effect.
Change of bandwidths shows a random and insignificant effect, which is very similar to that of kernel change effect.
Therefore, we cannot say that applying different bandwidths has a certain effect on N-ABE performance.
%As it was the case for different kernels, different bandwidths also generate only small variations.

\textit{RQ4. How do the characteristics of software effort datasets influence the performance of kernel weighting in N-ABE?}
%Kernel density estimation is a non-parametric method and reported to yield promising results in various data types~\cite{Frank03, John1995, Palpanas2003}.
%However, we did not see a single setting in our research that would suggest a similar promising result.
%In this paper we reported different results than the previous studies that used kernel estimation.
%kernel density estimation for N-ABE falls short of improving ABE0 in our experimental settings.
%This fact can be attributed to the particular characteristics of software effort datasets.
Effort datasets are much smaller than most of the datasets in different domains.
%This is due to the fact that each instance in an effort dataset is a completed project, which may take multiple years to be completed.
The dependent variable (effort value of a completed project) is highly variable.
%In other words, completion time of projects may diverge significanlty.
Furthermore, the attribute values are very open to personal judgment and error.
All these factors suggest that non-parametric methods may be failing due to inherent characteristics of software effort data.
%Although it may be true that kernel weighted N-ABE methods' low performance is due to inappropriate data characteristics of effort data, we still fail to answer why we observe the same low performance for IRWM weighted N-ABE.
%Therefore, with the available data at hand, we can not make a decisive conclusion on the effects of data characteristics on kernel weighting for N-ABE.


%\textit{RQ5.  Which parameters among kernel, bandwidth and selected analogy number matter the most in terms of estimation accuracy?}
%%Our results showed that in a N-ABE model with kernel density estimation neither kernel nor the bandwidth matters.
%Unlike change of kernel and bandwidth that did not provide any noticable difference in terms of accuracy values, \textit{k} value influenced performance measures considerably and we always observed higher accuracies for lower \textit{k} values.
%Therefore, among the three, the selected number of analogies (\textit{k}) matters the most.

\section{Future Work}
\label{sec:future-work}

We can identify 3 main domains in which this study may be extended:
\begin{enumerate}
\item \textit{Dataset:} In this research we used 19 software effort datasets. 
However, this study may be replicated on other software effort datasets as well.
%Furthermore, ABE is not restricted to software effort estimation domain.
%Kernel density estimation may be experimented on other ABE-applicable datasets as well.
\item \textit{Weighting Strategy:} 
%Strategy for non-uniform weighting in a N-ABE method may be completely different than kernel density estimation.
Another future direction can be experimentation on different non-uniform weighting strategies that are preferably based on different assumptions.
%\item \textit{Kernel Type:} We used 5 kernels (including IRWM) as non-uniform weighting strategies.
%But the ones used in this research are obviously not the only kernel types.
%%It may be the case that other particular kernels will perform differently than the ones used here.
%One future direction to this research could be the investigation of different kernels for better performance.
\item \textit{Customization:} Experimentation in this paper is based on publicly available datasets.
Our knowledge about the context and assumptions of these datasets are limited to their documentation. 
One further direction to this research could be customizing this work for a particular company and derive the actual problems associated with non-uniform weighting schemes.
\end{enumerate}

The experiments shown in this research took three months to research, design, execute, then write up.
It turns out that we could have spent the time more productively on other issues. 
Our pre-experimental intuition that ``non-uniform weighting in the data sparse domain of software effort estimation may not provide an improvement in estimation accuracy'' turned out to be correct. 
We want to remind researchers, who want to follow afore mentioned future directions that those future directions may as well end up in negative results.

%One of the future directions to this research would be to use different weighting strategies, which would better understand the datasets.
%Another future work could be to understand the reasons of small variations coming from different kernel-bandwidth combinations.
%One very obvious future work to follow could be the use of different kernels or bandwidths for weighting.
%However, depending on our results we do not recommend the last future work as a promising one.



\bibliographystyle{abbrv}
\bibliography{myref}

\clearpage
\input{replies-to-reviewers}

\end{document}

