\documentclass{svjour3}                     % onecolumn (standard format)
%\documentclass[smallextended]{svjour3}     % onecolumn (second format)
%\documentclass[twocolumn]{svjour3}         % twocolumn
%
\smartqed  % flush right qed marks, e.g. at end of proof
%
\usepackage{graphicx}
\usepackage{verbatim}
\usepackage{fancyvrb}
\usepackage{algorithmic}
\usepackage{cite}
\usepackage{multirow}
\usepackage{rotating}
\usepackage{subfigure}
\usepackage{float}
\restylefloat{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newcommand{\bi}{\begin{itemize}}
\newcommand{\ei}{\end{itemize}}
\newcommand{\be}{\begin{enumerate}}
\newcommand{\ee}{\end{enumerate}}
\newcommand{\tion}[1]{\S\ref{tion:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{itemize}
   \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{itemize}}
\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}

\begin{document}

\section{Supplementary Results}
\label{sec:results}

%As we mentioned earlier, we will evaluate the effect of non-uniform weighting against uniform weighting in an ABE context according to three performance measures: MdMRE, Pred(25) and Win-tie-loss values.
%In this section we present our results.
We first evaluate the MdMRE and Pred(25) results for each dataset and then present the win-tie-loss values. 
In this paper, we include only Cocomo81 results due to 2 reasons: 1) We have strict page restrictions  and result tables consume too much space, 2) observations from one dataset are valid for other datasets, i.e. non-uniform weighting never improves standad ABE.
For other tables please contact one of the authors or see 
 
\subsection{Evaluation of AR, MdMRE and Pred(25) Results}
\label{subsec:mre-based-results}

\subsubsection{Results for Nasa93}
Figure \ref{fig:mre-nasa93} lists the MdMRE as well as Pred(25) results for Nasa93 dataset.
As we can see from Figure \ref{fig:mre-nasa93}, different kernel types generate very similar results for various \textit{k} values. 
In other words, change of kernel does not have a considerable effect on the performance of N-ABE.
Furthermore, small performance changes due to different kernels do not follow a particular pattern.  

Like the change of kernel, changing bandwidth has almost a non-existent effect.
We see in Figure \ref{fig:mre-nasa93} that different bandwidths generate very close MdMRE and Pred(25) results for N-ABE methods.
More importantly there is no observable pattern in small changes due to kernel or bandwidth alterations.
Another property of Nasa93 results is that uniform weighting in ABE0 gains higher estimation accuracies (lower MdMRE and higher Pred(25) values) than corresponding N-ABE methods.
Lastly, we need to mention that similar to our use of Cocomo81 dataset, for Nasa93 we use all 93 instances and all 17 features in our treatments as well.

\input{figmrenasa93}


\subsubsection{Results for Desharnais}
We provide the MdMRE and Pred(25) values for Desharnais dataset in Figure \ref{fig:mre-desharnais}.
Among all kernel-bandwidth combinations we do not see a case where use of non-uniform weighting improves the performance of uniform weighting.
Therefore, particular characteristic of previous datasets in terms of being indifferent to different kernel methods is also valid for Desharnais dataset.
Furthermore, what we see from Figure \ref{fig:mre-desharnais} is that instead of improving ABE0 methods, N-ABE methods generate considerably worse MdMRE and Pred(25) results.
In the experiments we used 81 instances from Desharnais dataset.
For missing attributes of 4 projects in Desharnais, simple mean imputation is employed.

\input{figmredesharnais}

\subsection{Evaluation of WIN-TIE-LOSS Results}

\subsubsection{Results for Nasa93}
Figure \ref{fig:win-tie-loss-nasa93} shows the win-tie-loss results for Nasa93 dataset.
The results for Nasa93 are extremely similar to Cocomo81, that is in all cases the highest $win$ values belong to $k=3$ and $tie$ values are usually around 25\% of 180 comparisons.
Furthermore, application of different kernels for N-ABE does not yield a considerable difference.
For instance, for the treatment $k=3$ and $h=1/sqrt(size)$ the difference between the highest and the lowest $win$ value ($141$ and $122$ respectively) is only $19$, which is around 10\% of all 180 comparisons.
Similar to the effect of changing kernels, changing bandwidth also falls short of providing any noticable increase or decrease in estimation performance.
Furthermore, we need to point out hat in none of the \textit{k} values have N-ABE methods provided any improvement in estimation accuracy.
This shows us that like Cocomo81 dataset, Nasa93 dataset does not favor non-uniform weighting over uniform weighting.

\input{figwintielossnasa93}

\subsubsection{Results for Desharnais}
The win-tie-loss values for our last dataset are given in Figure \ref{fig:win-tie-loss-desharnais}.
The interpretation of Figure \ref{fig:win-tie-loss-desharnais} shows us a similar scenario to previous two datasets: Highest $win$ values were attained by $k=3$ and the treatments are statistically different from one another for most of the cases.
Furthermore, just like the Cocomo81 and Nasa93 datasets, the effect of different kernels as well as the effect of various bandwidths are negligible and do not follow a certain pattern.
Another similarity is that in none of the kernel-bandwidth combinations has N-ABE yielded higher estimation performance than ABE0.

\input{figwintielossdesharnais}



\bibliographystyle{abbrv}
\bibliography{myref}

\end{document}