\section{Results}
\label{sec:results}

To see the effect of kernel weighting, rigorous experimentation on 19 datasets and 3 different performance measures was performed.
Also, for each performance measure we tried 4 different kernels subject to 5 different bandwidths, plus the IRWM kernel (which does not have a bandwidth concept) and reported associated $win,tie,loss$ statistics.

\input{sampleTable}

\fig{win-tie-loss-desharnais} is a sample table, where we report the $win,tie,loss$ 
statistics of Desharnais dataset for ABE0 and N-ABE through Gaussian kernel. 
For each dataset we have $4$ such tables (one for each kernel), so for all datasets there is \textit{19 Datasets $\times$ 4 tables = 76 tables}.
For the IRWM kernel there will be another \textit{19 datasets $\times$ 1 kernel = 19 tables}.
So in total, we have \textit{76 + 19 = 95 tables} to report, which is impractical to put in a summary report such as this one.
Therefore, we provide the sample table of \fig{win-tie-loss-desharnais} as a pointer and manual to other 94 tables, which are given under \cite{supplementaryResults}.

In \fig{win-tie-loss-desharnais} we see that each row reports $win,tie,loss$ statistics of ABE0 methods (\textit{k=3,5,7,9,best}) as well as N-ABE methods (\textit{k=[3,5,7,9,best]+kern where kern stands for kernel weighting}) subject to a particular performance measure.
Similarly, each column shows the $win,tie,loss$ statistics associated with a particular bandwidth value.
As can be seen in \fig{win-tie-loss-desharnais}, ABE0 methods always have higher $win$ values and always have a $loss$ value of $0$, meaning that they never lose against N-ABE methods.
By analyzing each row/column intersection we can see that the performance of ABE0 is never improved by N-ABE.

The analysis of \fig{win-tie-loss-desharnais} is only valid for Desharnais dataset and Gaussian kernel.
So as to provide an executive summary of all 19 datasets and all kernel/bandwidth combinations, we will use \fig{summary-all}.
Each row of \fig{summary-all} shows the comparison of ABE0 performance to that of N-ABE subject to 3 different performance measures.
Every kernel/bandwidth intersection in \fig{summary-all} has 3 symbols corresponding to MdMRE, MAR and Pred(25) comparisons from left to right.
Each of these 3 symbols can have 3 values: $-,+,o$.
A \textit{``$-$''} symbol means that N-ABE decreased the accuracy of ABE0, whereas a \textit{``$o$''} symbol means ABE0 and N-ABE are statistically same in all conditions and a \textit{``$+$''} symbol shows that ABE0 accuracy was improved through kernel weighting (i.e. N-ABE has a better accuracy than ABE0).

So as to decide if ABE0 performance is improved ($+$), decreased ($-$) or not changed ($o$) by N-ABE we look at all of the $95$ detail-tables such \fig{win-tie-loss-desharnais} and see the behavior of 5 \textit{k} values under ABE0 and N-ABE.
If the majority of \textit{k} values (at least 3 out of 5) are improved by N-ABE in terms of $win-loss$ values, then we assign a ``$+$'' symbol for that setting; if the performance associated with the majority of \textit{k} values is decreased by N-ABE, then we assign a ``$-$'' symbol; otherwise, we assign a ``$o$'' symbol to that setting.

See in \fig{summary-all} that out of $19$ datasets, there is only one dataset (SDR) where N-ABE provides a performance improvement in certain cases. 
Even for that dataset there are $15$ ``$+$'' symbols and $21$ ``$-$'' symbols, meaning that most of the time N-ABE is still destructive.
In $18$ datasets, which is $\frac{18}{19} = 95\%$ of all the datasets, there is not a single case where N-ABE improves the performance of ABE0.
Also note that \fig{summary-all} provides an extensive summary of different error measures (MdMRE, MAR, Pred(25)) as well as kernels and bandwidths.
Therefore, our conclusion from \fig{summary-all} that \textit{``non-uniform weighting through standard kernel methods does not improve the performance of ABE''} is quite stable accross different datasets and error measures.

\input{allDatasetResults}

\input{irwm-results}

Another summary table is given in \fig{summary-irwm}.
\fig{summary-irwm} is very similar to \fig{summary-all} in the sense that it summarizes the performance of N-ABE over 19 datasets w.r.t. three different performance measures.
The difference is that \fig{summary-all} summarizes the results of standard kernel methods, whereas in \fig{summary-irwm} we see the N-ABE performance under an \textit{expert-based} kernel, i.e. IRWM.
Although there are important differences between standard and expert-based kernels (IRWM has no bandwidth parameter), the results seen in \fig{summary-irwm} is quite similar to those of \fig{summary-all}.
As can be seen in \fig{summary-irwm}, there is not a single case where N-ABE (under IRWM kernel) improves the performance of ABE0.
Furthermore, the amount of ``$-$'' symbols is much more than ``$o$'', meaning that N-ABE decreases the performance of ABE0 most of the time.

%As we mentioned earlier, we will evaluate the effect of non-uniform weighting against uniform weighting in an ABE context according to three performance measures: MdMRE, Pred(25) and Win-tie-loss values.
%In this section we present our results.
%We first evaluate MAR, MdMRE and Pred(25) results for each dataset and then present related win-tie-loss values. 
%In this paper, for standard kernel methods only Cocomo81 results are included due to 2 reasons: 1) We have strict page restrictions  and result tables consume too much space, 2) all observations from Cocomo81 are valid for other datasets, i.e. non-uniform weighting never improves standad ABE.
%For not-included tables please contact the authors or see \cite{supplementaryResults}.
 
%\subsection{Evaluation of MAR, MdMRE and Pred(25) Results}
%\label{subsec:mre-based-results}
%
%%The MRE-based measures we investigate in this section are MdMRE and MMRE.
%The MAR, MdMRE and Pred(25) values for Cocomo81 is provided in Figure \ref{fig:mre-coc81}.
%In this Figure N-ABE settings are indicated with a $+N$ sign, where $+N$ stands for \textbf{\textit{N}}on-uniform weighting.
%Furthermore, different number of analogies used in each treatment are represented by actual \textit{k} values, whereas \textit{\textbf{d}ynamic k} is represented by a $d$ symbol.
%
%
%\subsubsection{Results for Cocomo81}
%In Figure \ref{fig:mre-coc81} we see MAR, MdMRE and Pred(25) values for Cocomo81.
%The first observation we can make out of Figure \ref{fig:mre-coc81} is that for the same \textit{k} value N-ABE methods fail to improve ABE0, i.e. kernel methods cannot provide a performance improvement.
%%For all settings, uniform weighting yields lower error rates.
%Application of different kernels as well as different bandwidths for non-uniform weighting does not produce a recognizable pattern in terms of MAR, MdMRE and Pred(25).
%%Change of bandwidths for kernels does not create a recognizable pattern in the results either.
%Another observation from Figure \ref{fig:mre-coc81} is that both for uniform and non-uniform weighting, smaller \textit{k} values yield lower MdMRE and Pred(25) values.
%%Regarding Figure \ref{fig:mre-coc81}, note that we use all $63$ instances of Cocomo81 dataset with full range of its features in all treatments.
%
%
%
%
%\subsubsection{IRWM Results for All Datasets}
%Figure \ref{fig:irwm-mdmre-mmre} presents MAR, MdMRE and Pred(25) results for IRWM kernel.
%The difference of results in Figure \ref{fig:irwm-mdmre-mmre} is that previous results of N-ABE methods came from standard kernel methods (minimum human interaction), whereas in Figure \ref{fig:irwm-mdmre-mmre}  weighting strategy is defined by human experts (complete human dependence).
%Despite the difference in the weighting strategies, the trend in the results are very alike, i.e. in none of the 3 datasets can N-ABE methods outperform ABE0.
%%Therefore, after 330 settings which involve both non-parametric methods and expert methods for weighting N-ABE, we still do not observe a case where N-ABE methods could bring an improvement on simple ABE0 method.
%
%
%
%\subsection{Evaluation of WIN-TIE-LOSS Results}
%
%%In this section we evaluate the performance of non-uniform weighting and uniform weighting according to  win-tie-loss values.
%%Since we have 10 settings for each kernel subject to 20 runs, the sum of $win$, $tie$ and $loss$ values can be at most 180 (\textit{(10 settings - 1 setting itself) * 20 runs = 180}).
%%The nice point of win-tie-loss calculation is that it does not only show us whether N-ABE with kernel density estimation provides an improvement to ABE0, but it also shows us the performance of a single method (ABE0 or N-ABE) in comparison to all other methods.
%
%\subsubsection{Results for Cocomo81}
%In Figure \ref{fig:win-tie-loss-coc81} the win-tie-loss values for Cocomo81 are given.
%Since we have 10 settings for each kernel subject to 20 runs, the sum of $win$, $tie$ and $loss$ values can be at most 180 (\textit{(10 settings - 1 setting itself) * 20 runs = 180}).
%For all settings, the $tie$ values are most of the time less than 45 (less than 25\% of all the comparisons), which means that in 75\% or more of the comparisons there is a statistical difference between the methods.
%%The first observation we can make is that smaller number of analogies have always attained higher $win$ values and lower $loss$ values.
%However, when ABE0 is compared to N-ABE methods for different \textit{k} values, N-ABE can never improve the $win$ values over ABE0.
%
%From Figure \ref{fig:win-tie-loss-coc81} we can also see the effect of applying different kernels and different bandwidths on the performance of N-ABE.
%In terms of kernels, there is not a considerable performance difference between different types.
%This is consistent with prior research, which reports that different kernels yield similar results~\cite{Cressie1993}.
%%For Cocomo81 dataset we observe that the same fact is also valid for software effort estimation data. 
%On the other hand, our results in terms of bandwidth is contradictory.
%Bandwidth is reported to be influential in different contexts~\cite{Cressie1993, Scott1992, Wand1994}.
%However, we are unable to observe a considerable effectdue to different bandwidths.
%%In Figure \ref{fig:win-tie-loss-coc81} the win-tie-loss values of different kernels when used with 5 different bandwidths are very similar.
%%In fact, for the uniform kernel the performance is completely identical between different bandwidths.
%Therefore, 
%%we can propose that software effort data behaves differently when compared to other data types, i.e. 
%unlike the performance on spatial data performance on software effort data is not improved by changes in bandwidth. 
%
%
%
%\subsubsection{IRWM Results for All Datasets}
%%Up to this point we have observed 315 different settings and saw that neither kernel nor the bandwidth change does have a considerable impact on the performance of N-ABE.
%%Furthermore, we found out that simple ABE0 approach yields higher performance measures in terms of win-tie-loss values.
%%However, standard kernels are not the only alternatives of non-uniform weighting.
%%Another N-ABE methods we use in this research is so called IRWM~\cite{Mendes2002, Mendes2003}.
%The win-tie-loss values of all $3$ datasets for IRWM weighted N-ABE are given in Figure \ref{fig:irwm-win-tie-loss}.
%Since IRWM is a different weighting approach than kernel density estimation, we do not have bandwidths to compare in that scenario.
%%On the other hand with IRWM results we can mutually compare the estimation performances of N-ABE and ABE0 approaches.
%Our reading from Figure \ref{fig:irwm-win-tie-loss} is that N-ABE methods cannot outperform ABE0 in any one of the three datasets.
%In other words, just like standard kernel weighted N-ABE, IRWM weighted N-ABE also fails to outperform ABE0.
%Therefore, in a total of 330 settings we see that N-ABE methods are unable to outperform the performance of simple ABE0 approach.

