% THIS IS SIGPROC-SP.TEX - VERSION 3.1
% WORKS WITH V3.2SP OF ACM_PROC_ARTICLE-SP.CLS
% APRIL 2009
%
% It is an example file showing how to use the 'acm_proc_article-sp.cls' V3.2SP
% LaTeX2e document class file for Conference Proceedings submissions.
% ----------------------------------------------------------------------------------------------------------------
% This .tex file (and associated .cls V3.2SP) *DOES NOT* produce:
%       1) The Permission Statement
%       2) The Conference (location) Info information
%       3) The Copyright Line with ACM data
%       4) Page numbering
% ---------------------------------------------------------------------------------------------------------------
% It is an example which *does* use the .bib file (from which the .bbl file
% is produced).
% REMEMBER HOWEVER: After having produced the .bbl file,
% and prior to final submission,
% you need to 'insert'  your .bbl file into your source .tex file so as to provide
% ONE 'self-contained' source file.
%
% Questions regarding SIGS should be sent to
% Adrienne Griscti ---> griscti@acm.org
%
% Questions/suggestions regarding the guidelines, .tex and .cls files, etc. to
% Gerald Murray ---> murray@hq.acm.org
%
% For tracking purposes - this is V3.1SP - APRIL 2009

\documentclass{acm_proc_article-sp}

%
\usepackage{graphicx}
\usepackage{verbatim}
\usepackage{fancyvrb}
\usepackage{algorithmic}
\usepackage{cite}
\usepackage{multirow}
\usepackage{rotating}
\usepackage{subfigure}
\usepackage{float}
\restylefloat{figure}
\usepackage{times}
\usepackage{algorithm}

\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{itemize}
  \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{itemize}}

\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
   \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}


\usepackage[table]{xcolor}
\usepackage{url,graphicx}
\newcommand{\G}{\cellcolor[rgb]{0.8,0.8,0.8}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\hyp}[1]{Hypothesis~\ref{hyp:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}

\newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
\newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}


\begin{document}

\title{Twenty First Century Software Effort Estimation Application Process}

\numberofauthors{2} 
%
% 1st. author
\author{
\alignauthor
		Jacky Keung\\
       \affaddr{Department of Computing }\\
       \affaddr{       The Hong Kong Polytechnic University  }\\
       \affaddr{Kowloon, Hong Kong}\\
       \email{Jacky.Keung@comp.polyu.edu.hk}
\alignauthor Ekrem Kocaguneli, Tim Menzies\\
       \affaddr{ Lane Department of Computer Science and Electrical Engineering}\\
       \affaddr{West Virginia University}\\
       \affaddr{Morgantown, WV 26505, USA}\\
       \email{ekocagun@mix.wvu.edu, tim@menzies.us}
}


\maketitle
\begin{abstract}

In this paper, we are summarising our experience in the research of cost estimation, showing important aspects of successful cost estimation experiments.

Adding the research building blocks, and given that we are able to determine conclusion stability, a set of more completed general estimation application process should be used.
\end{abstract}

In the past :

Dataset -> Effort Estimator -> predictions

After a few years later, preprocessors such as Analogy-X, sensitivity analysis, feature subset selection etc surfaced

Dataset -> Preprocessor  ->  Effort Estimator -> predictions

Now we have a method (ASE Journal) for selecting the best combination of preprocessor + effort estimator

Dataset -> Method Selection -> Preprocessor (inc feature selection) -> Effort Estimator -> predictions 


The paper is a summary and provides an up-to-date for the latest development of effort estimation research, this will help us to put our work in a leading stage. 




This paper shows that we can propose a strong/weak relationship between datasets...
Many datasets used by prior publications are very  limited in number
to distinguish {\em strong/weak} datasets.

%In \fig{error_datasets}, we report percentage of {\em losses} for each error measure separately, instead of the {\em losses} reported above.
%The maximum number of wins for any dataset over ninety algorithms per error measure is $89\times90=8,010$. 
%\fig{error_datasets} sorts all 20 data sets by their total wins
%in all 
%seven performance criteria separately (expressed as a ratio of $8,010$). For example,
%with 
%the TELECOM dataset, all 90 methods rarely won.
Similarly, the maximum number of losses for any dataset over ninety algorithms is $89\times7\times90=56,070$. 
\fig{error_datasets} sorts all 20 data sets by their total losses
in all 
seven performance criteria (expressed as a ratio of 50,070). For example,
with 
the TELECOM dataset, all 90 methods rarely lost.

%\begin{figure}[!b]
%\begin{center} \includegraphics[width=3in]{lib/cols-win-all-together.pdf} \end{center}
%\caption{Total wins seen in 20 datasets, expressed as a percentage
%of the maximum number of possible wins seen for one datasets (so 100\%=8,010).}\label{fig:error_datasets}
%\end{figure}

%\begin{figure}[!th]
%\begin{center} \includegraphics[width=3in]{lib/cols.pdf} \end{center}
%\caption{Total losses seen in 20 datasets, expressed as a percentage
%of the maximum number of possible losses seen for one datasets (so 100\%=50,070).}\label{fig:error_datasets}
%\end{figure} 


\fig{rank_changes_dataset} is somewhat a continuation of \fig{error_datasets}, in the sense that it deals with the stability of datasests.
To test the stability, we question the mean of maximum rank change among datasets, when sorted w.r.t. $win$, $tie$, $win-loss$ over 7 error measures.
\fig{rank_changes_dataset} shows that the maximum value of mean-rank change is $18$, i.e. a method ranked as $2^{nd}$ in one scenario can rank as $20^{th}$ in another scenario.
Therefore, with that amount of datasets, it is not healthy to propose {\em strong} or {\em weak} datasets that always attain lowest/highest performance values.
If a dataset can change its position with a $+x$ or $-x$ amount, then there is a need for a window size of at least $2x$ and possibly some more datasets to actually observe how datasets would rank.

%\fig{rank_changes_dataset} distinguishes 3 regions when we impose a less than 10 mean rank change line.
%In these 3 regions, only $r2$ has datasets with less than 10 mean rank change.
%$r2$ contains 9 different datasets from cocomo81o to desharnais.
%When we impose these 3 regions onto \fig{error_datasets}, we see that stable datasets fall between the loss percentages of $5\%$ to $15\%$ (see dashed lines).

%\begin{figure}[!th]
%\begin{center} \includegraphics[width=0.7\textwidth]{lib/rank-changes-dataset.pdf} \end{center}
%\caption{Datasets and the mean of their maximum rank changes over all performance measures w.r.t. $win$, $loss$ and $win-loss$ values. Some datasets have lower rank-changes. However, the maximum mean-rank change is around $18$ and we need more than $2 \times 18 = 36$ datasets to claim an order, hence strong/weak rekationship, between datasets.}
%\label{fig:rank_changes_dataset}
%\end{figure}


Our  datasets could be sorted according to how well
they can distinguish between  effort estimators; for that matter, there is a need for more publicly available datasets.
%specifically, eleven
%datasets (in common use  in the effort estimation literature)   are
%{\em weak}; i.e. poorly distinguish the behavior of different
%estimators. 


\bibliographystyle{abbrv}
\bibliography{icssp,timm}

\end{document}