\documentclass{acm_proc_article-sp}

\usepackage{stfloats}
\usepackage{times}
\usepackage{wrapfig}
\usepackage{cite}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{multirow}
\usepackage{rotating}
\usepackage{subfigure}


%\usepackage[none]{hyphenat} 
\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}%
  \setlength{\partopsep}{0pt}%
  \setlength{\parskip}{0pt}%
  \begin{itemize}%
  \setlength{\leftmargin}{.2in}%
  \setlength{\parsep}{0pt}%
  \setlength{\parskip}{0pt}%
  \setlength{\itemsep}{0pt}}%
 {\end{itemize}}

\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
   \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}


\usepackage[table]{xcolor}
\usepackage{url,graphicx}
\newcommand{\G}{\cellcolor[rgb]{0.8,0.8,0.8}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\hyp}[1]{Hypothesis~\ref{hyp:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}

\newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
\newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}

\begin{document}

\title{How to Find Relevant Data for Effort Estimation?} 


\numberofauthors{1} %  in this sample file, there are a *total*
% of EIGHT authors. SIX appear on the 'first-page' (for formatting
% reasons) and the remaining two appear in the \additionalauthors section.
%
\author{
% 1st. author
\alignauthor
Ekrem Kocaguneli, Tim Menzies\\
%	   \titlenote{Dr.~Trovato insisted his name be first.}\\
       \affaddr{Lane Department of Computer Science and Electrical Engineering}\\
       \affaddr{West Virginia University,Morgantown, USA}\\
       \email{ekocagun@mix.wvu.edu, tim@menzies.us}}

%\date{30 July 1999}
% Just remember to make sure that the TOTAL number of authors
% is the number that will appear on the first page PLUS the
% number that will appear in the \additionalauthors section.

\maketitle
\begin{abstract}
~\\
\textit{\textbf{Background}}: Building effort estimators requires
the correct selection of training data.  How can we find that data?
It is tempting to \textit{cross} the boundaries
of development type, location, language, application and hardware
to use existing datasets of other organizations. However,
prior results caution that using such {\em cross} data may not be useful. \\
\textit{\textbf{Aim}:}  We test the conjecture that {\em instance
selection} can automatically prune irrelevant instances and retrieval from
the remaining examples is useful for effort estimation, regardless
of  their source.\\
\textit{\textbf{Method}:} We selected 8 \textit{cross-within
divisions} (21 pairs of \textit{within-cross} subsets) out of 19
datasets and evaluated these divisions under different analogy-based
estimation (ABE) methods.
\\
\textit{\textbf{Results}:} 
Between the {\em within} and {\em cross} experiments,
there were very few
statistically significant differences in (a) the  performance of effort estimators;
or (b) the amount of instances
retrieved for estimation.
\\
\textit{\textbf{Conclusion}:}
There is no practical difference between {\em cross} and {\em within} data.
After applying instance selection,
the remaining
examples, be they from \textit{within} or from \textit{cross} source divisions, can be used
for effort estimation.
\end{abstract}


\category{H.4}{Software Cost Estimation}{\textit{k}-NN}
\category{D.2.8}{Software Engineering}{Cost}[within resource, cross resource]

\newtheorem{fore}{{\bf PREDICTION}}


\section{Introduction}
A recurring problem in effort estimation is finding training data
that is relevant to some local problem. 
%Our current training sets
%are often too small to gaurentee that our learners have seen enough data:
%\bi
%\item
%Orrego~\cite{orrego04}
%documented where training  {\em stablizes}; i.e. where
%learning from $N$ examples is not better than learning from $M<N$.
%For models built by linear regression from 10 data sets with one to dozen variables,
%stabilization occured after 25,25,30,50,50, 80,100,100,150,300 examples.
%\item
%The size of publically available
%data sets often much smaller than Orrego's stabilization points:
%e.g. the median number of examples by
%Mendes et al.~\cite{Mendes2003}, Auer et al.~\cite{Auer2006},
%Baker~\cite{baker07}, this study, and Li et al.~\cite{Li2009} was
%13,15,31,33,52.
%\ei
When we cannot find enough local training data, it is tempting to 
try and import data from other sources. However, it is not
clear that this approach is useful: many studies report that using imported data
degrades estimation efficacy, perhaps because the imported data is not relevant to
the local context (e.g. see the
Kitchenham  et al.~\cite{Kitchenham2007} and
Zimmermann et al.~\cite{Zimmermann2009} studies discussed later in this paper).

In this paper, we offer one solution to the problem of importing relevant data from 
other sources in order to make estimates about local models. Our solution is based
on a fresh look at what it means to say that examples are {\em local} or {\em imported}.
Many
publications~\cite{stukes98,stukes91,stukes99,ferens98,habib98,lum02,boehm00a}
including several of our own~\cite{Menzies2005,Menzies2006} either
explicatively or tacitly assume ``$locality(1)$''; i.e.  clumps of
similar projects can be discovered {\em using a single  feature}.  For
example, some authors claim that, for a project in a specific
organization, software effort models work better when calibrated with
local data collected within that same organization. Proponents of such
a {\em within source} approach assume that it is best to retrieve training
data for examples divided acccording to:
\bi
\item
The {\em project type} being developed: e.g. embedded, etc;
\item
The {\em development centers}  of the different developers;
\item
The {\em development language} of the projects;
\item
The {\em application type} (management information system;
guidance, navigation, and control; etc);
\item
The {\em targeted hardware platform};
\item
Just the {\em in-house or outsourced} development projects;
\ei

It would be useful if $locality(1)$
were false since, if true,
this means that any lessons learned
from one organization may never apply to another.
If so,
then our ability to
make general conclusions about software engineering (SE) would
be confined to small,
highly specialized,
sub-groups (e.g. just one company).

The opposite to $locality(1)$ is 
``$locality(N)$''; i.e. the assumption that
effort estimation data  forms a  complex multi-dimensional space that can only
be  usefully divided {\em using multiple features}.
If this were true, then this would be very good news since that would mean that
relevant data for effort estimation does not come just from
small sub-groups within one organization. Rather,  useful data could be collected
from many projects including \textit{cross sources}. 
This would simplify the construction of effort models for new projects:
just search other contexts for the right data 
for the new project.
Also, it could lead to conclusions about SE
that are general to many  development contexts.
This paper argues against $locality(1)$ and for $locality(N)$ using
two predictions that would support $locality(N)$ and would contradict $locality(1)$:


{\bf PREDICTION 1:} {\em Effort models built from training data divided on a single  feature
will perform no better (and, perhaps, even worse) than those that divide the data
using multiple features.}


{\bf PREDICTION 2:} {\em Consider project data that was grouped into divisions
w.r.t. to the value of a feature.
If training data is retrieved from within and across those divisions,
then it would be $equally$ as probable to find useful data {\em within} as {\em cross} those
divisions.}\\

Recent research offers much support for
{\bf PREDICTION~1}.
In a study
of 90 effort estimation methods (ten pre-processors $\times$  nine
learners),
we found that the best methods
were those that divided the training data according to multiple features~\cite{Keung2011}.
That result is detailed in our {\em Related Work}
section.

For {\bf PREDICTION~2},
we test if  divisions
based on different {\em single} features change the  effort estimation process.  
We allow an {\em instance selection \& retrieval} algorithm to find which instances are best
for training. That algorithm is given access to all the training data, or
just the data divided via a single feature.
It  will be shown that:
\begin{quote} {\em 
The probability of retrieving
training data from within or across divisions based on single features \underline{is the same}.}
\end{quote}
This result, plus the results in~\cite{Keung2011}, are strong support for $locality(N)$
since they confirm both {\bf PREDICTION~1} and {\bf PREDICTION~2}.
These results have three practical implications:
Firstly, it would mean that
effort estimation is not dependent on some artificial, and trite,
 division of training data such as 
the source organization; the kind of application; or any other single feature
division. 
Given the complex multi-dimensional nature of the software creation process, 
divisions of training data based on (say) geographical dimensions may be less important than other factors. 
The most similar software to what you are writing now may not be in the next office. 
Rather, it may be in an office on the other side of the world 
(and automatic {\em instance
selection \& retrieval algorithms} can  find that data).

Secondly, if $locality(N)$ was true,
it would be useful to build effort estimation models from data taken from multiple contexts.
Hence:
\bi
\item Building estimation models is less expensive
since generating local models need not wait for an elaborate
(and expensive and time-consuming) collection process from local data.
\item
There are effects in SE that transcend our current divisions
of data (e.g. data from company1 or company2). This is a very exciting result since it
promises future generalizable SE results that apply to many organizations.
\ei
Thirdly, $locality(N$)
offers a strong business case for collecting SE data in some sharable repository.
Such repositories can now be trusted to provide, at least some, relevant historical examples
for building effort estimators for some new project.

\subsection{Terminology}

Here, we offer some terminology clarification.
Other papers such as
Kitchenham  et al.~\cite{Kitchenham2007} and
Zimmermann et al.~\cite{Zimmermann2009} 
discuss the impact of dividing data according to a single
feature (either 
the {\em organization} or {\em application} name).
This paper explores those single-feature division {\em as well as} other single feature divisions such as the {\em project type} being developed; the {\em development centers}
of the developers; the \textit{development language} of the software; the  {\em application type}; the {\em targeted hardware platform}; and \textit{in-house or outsourced} development. 
So whereas (e.g.) Kitchenham et al. explore cross-vs-within company data,
we explore cross-vs-within data ``{\em sources}''  where the data is divided by the values
of \textit{any} single feature.
As shown below, we find no examples where single feature division improves a state-of-the-art
effort estimation method.

Other two terms that need clarification are \textit{instance selection} and \textit{instance retrieval}.
Instance selection refers to a filtering method, where data is filtered to a subset of more \textit{relevant} instances; hence, \textit{instance selection} and \textit{filtering} is used interchangeably in the rest of the text.
On the other hand, \textit{instance retrieval} is the process of finding the closest neighbors (e.g. in a \textit{k}-NN method).
In that sense, \textit{instance selection} is a pre-processor to a second process; whereas, \textit{instance retrieval} is the part of a single estimation process.

\section{Related Work}
\subsection{Evidence for PREDICTION 1}

Keung et al.~\cite{Keung2011} built 90 effort estimators using
10 pre-processors and
9 learners.
Pre-processors included normalization, various discretization methods and
feature selection (PCA, stepwise, sequential forward). 
Learners included  $k=1$ and $k=5$-nearest-neighbor, linear and stepwise regression, 
CART, neural nets and PCR.

The 90 estimators were assessed via multiple accuracy statistics.
Let  $t$ instances have actuals $a_1,a_2,..,a_t$.
Prediction models generate prediction $p_i$ for  instance $i$.
If $|p_i - a_i|$ is $AR_i$ (the  {\em absolute residual}  difference between  predictions and actuals)
then:

\bi
\item
$MAR$ is the {\em mean absolute residual} $(\sum_i p_i - a_i)/t$.
\item
$MRE_i$ and $MER_i$ are the ratios  
$AR_i/a_i$ and 
$AR_i/p_i$.
\item
$PRED(X)$ is the percent of $t$ with $MRE_i \le X\%$.
\item
$MMRE =\sum_i MRE_i/t$
\item
$MMER = \sum_i MER_i/t$.
\item
$MdMRE$ is median $MRE_i$ value. 
\item
Finally, the {\em balanced errors} are\newline
\mbox{$MBRE_i = (p_i - a_i )/ min(p_i,a_i)$} and\newline 
\mbox{$MIBRE_i = (p_i - a_i )/ max(p_i,a_i)$}
\ei

The 90 estimators were used in a leave-one-out study
on  twenty data sets from http://promisedata.org/?cat=14, and were compared via a Mann-Whitney test (95\% confidence).
As it might have been predicted by Shepperd et
al.~\cite{Shepperd2002}, the ranking of the estimators varied
across different data sets and the different accuracy estimators.

However, Keung et al. found a small group of 13 estimators that were
consistently the best performers across all data sets (measured
according to all of MAR, MRE, PRED(25), MMRE, MMER, MdMRE, MBRE, and
MIBRE).  
In terms of this paper, 
the major result of Keung et al. is that all these 13 estimators
used CART or $k=1$ nearest neighbor. This is significant since
both these estimators use multiple features to sub-divide the training data:
\bi
\item
$k$-th nearest neighbor algorithms use all project features (perhaps, weighted by some
feature) to determine related projects~\cite{Shepperd1997};
\item
Tree-based algorithms like CART~\cite{Breiman1984} divide data into multiple
branches,  where each branch tests and divides that data on
multiple features.
\ei
Hence, this result is  strong support for {\bf PREDICTION~1}.


\subsection{Other Evidence}

Other results in the literature are also inconclusive about  $locality(N)$.
In their review of papers building effort models using data from within one
company or across
multiple companies, Kitchenham et al.~\cite{Kitchenham2007} found  equal evidence for and against the value of
building effort models based on a single feature division (specifically, they
found four studies favoring the use of \textit{within} company
data, and another three reporting that  using
\textit{cross} data performance is not significantly worse than \textit{within}). 
In other work, in the field of defect prediction,
Zimmermann et al.~\cite{Zimmermann2009} found that predictors
performed worse when trained from cross-application data than from within-application data. The evidence for their conclusion is quite emphatic:
{\em within} defeated {\em cross} in
618 out of 622 comparisons.

On the other hand, support for $locality(N)$
comes from the work of Turhan et al.~\cite{Turhan2009},
and 
Kocaguneli et al.~\cite{Kocaguneli2010}.
Turhan et al. compare defect predictors learned from cross or within resource data.
Like Zimmermann, they found that using {\em all} cross resource data lead to poor predictor
performance (very large false alarm rates). However, after {\em instance selection} pruned
away irrelevant cross resource data, they found that the cross resource predictors were
equivalent to the predictors learned from within resource data~\cite{Turhan2009}.
Inspired by ~\cite{Turhan2009},
Kocaguneli et al.~\cite{Kocaguneli2010} used {\em instance selection}
as a pre-processor for a study on cross-vs-within resource effort estimation.
In a limited study with
three data sets, they found that after instance selection, 
the performance differences in
the predictors learned from cross or within data were statistically insignificant.

\subsection{Resolving the Evidence}

The results in~\cite{Turhan2009} and \cite{Kocaguneli2010} support $locality(N)$ but the other
results discussed above are inconsistent or unsupportive.
How can we reconcile this conflicting evidence?
One way is to note that:
\bi
\item Studies supporting $locality(N)$ all used a filtering method (instance selection).
\item Instance selection is {\em not} seen in the
Kitchenham, Zimmermann et al. studies.
\ei
%The one exception is that 
%one study surveyed by  Kitchenham et al. explored  analogy-based estimation
%and, in that analogy-based result, training on cross resource data worked as well as 
%training on within resource data. 
An \textit{instance selection} method uses every feature (perhaps, with
some feature weighting) to find relevant training examples.
Hence, the studies with \textit{instance selection} \cite{Turhan2009, Kocaguneli2010}  offer more  support for  $locality(N)$ than for $locality(1)$; however, they
are hardly conclusive, since they do not collect the information required to
comment on {\bf PREDICTION~2}. What is required is a well-controlled
 instance selection and retrieval experiment over 
data divided by some single feature. {\bf PREDICTION~2} would be supported
if the instance selection \& retrieval method (TEAK) retrieved as much data {\em within} as {\em across} the filtered single feature divisions. 

\begin{figure}[!t]
  \centering                
  \subfigure[\textit{Within}]{\label{fig:within-problem}\includegraphics[height=6cm, width=1.5cm]{withinProblem-2.png}}~~
  \subfigure[\textit{Cross}]{\label{fig:cross-problem}\includegraphics[height=6cm, width=1.5cm]{crossProblem-2.png}} ~~
  \subfigure[\textit{Filter}]{\label{fig:cross-filter-problem}\includegraphics[height=6cm, width=1.5cm]{crossFilterProblem-2.png}} ~~
  \subfigure[\textit{ Within \& Cross Filter}]{\label{fig:within-cross-problem}\includegraphics[height=6cm, width=1.8cm]{withinCrossProblem-2.png}} 
  \caption{The problem types in \textit{within} vs. \textit{cross} data comparison and our conclusions so far. \textit{``+''} and \textit{``-''} signs on top of each approach
 mean positive and negative results respectively. A \textit{``?''} sign means that
the approach  has not previously been investigated.}
  \label{fig:problem-types}
\end{figure}

%\newpage

\section{Methodology}
\fig{problem-types} illustrates how this study differs to prior work.
Most effort estimation research falls into \fig{problem-types}.a where 
estimation models are applied {\em within} one source set to learn an estimator.
Examples of this approach include \cite{Mendes2003,Auer2006,Li2009,Shepperd1997,Menzies2006}.

A smaller number of papers, such as those surveyed by Kitchenham et al.~\cite{Kitchenham2007},
explore building models using data that falls {\em across} many sources (see \fig{problem-types}.b).
Fewer still are the papers like ~\cite{Turhan2009,Kocaguneli2010}) that, prior to learning,
apply some instance selection to cross resource data
sources (see \fig{problem-types}.c).

To the best of our knowledge, this paper is the first that allows an effort estimator 
to select (filter) training data from  either {\em cross} or {\em within} different sources,
then checks what data was retrieved from which source  (see \fig{problem-types}.d).


%The details of dataset filtering and division of them into \textit{cross-within divisions} are presented below.
%We also explain the selected performance measures and our experimental set-up.

\subsection{Datasets}
There are 2 fundamental factors that were considered for selection of the datasets used in this research:
\bi
\item Public availability: For reproducibility purposes
\item Cross-within divisibility: For enabling \textit{cross} vs. \textit{within} experimentation
\ei
A critical issue in SE is the ability of the proposed results to be reproducible~\cite{Juristo2009, Kitchenham2007} and use of proprietary data is a major obstacle towards this goal.
Therefore, all our datasets are publicly available through PROMISE data repository~\cite{Boetticher2007}.

We define \textit{cross-within division} as the subset(s) of effort
data that are formed through division of one feature:
instances having the same value for that feature  form a
subset.  
Such features are plausible candidates for generating 
a \textit{cross} source experiment, i.e. the features should be
likely to change from one source to other.
Accordingly,
this study began by exploring what PROMISE effort data
can be divided via a single feature.
After manually inspecting more than 20 datasets,
six were found to be suitable for \textit{cross-within} experimentation.
Those six data sets support the
21
\textit{cross-within divisions} shown in \fig{cc-selection}.
The selected division criteria include:
\bi
\item project type: embedded, organic and semidetached (\textit{cocomo81}), 
\item center: geographical development center (\textit{nasa93}), 
\item language type: programming language used for development (\textit{desharnais}), 
\item application type: on-line service program, production control program etc. (\textit{finnish} and \textit{maxwell}), 
\item hardware: PC, mainframe, networked etc. (\textit{kemerer} and maxwell), 
\item source: whether in-house or outsourced (maxwell).
\ei
We will use the following nomenclatures:
If a subset name is followed by a set of numbers, they correspond to values of the  feature used to form the subset.
If a name has multiple numbers at the end (e.g. finnishAppType2345) then all instances with these values are combined in a single subset.
%Also note that the terms \textit{within/cross} and WS/CS will be used interchangeably for the rest of the text.

\begin{figure}
\scriptsize
\center
\begin{tabular}{l | l | l | r}
\textbf{Dataset} & \textbf{Criterion} & \textbf{Subsets} & \textbf{Subsets Size}\\[1ex]\hline\hline
cocomo81 & project type & cocomo81e &	28  \\
  & & cocomo81o   & 	24  \\
  & & cocomo81s   & 	11	  \\\hline
  nasa93 & development center & nasa93\_center\_1   & 	12  \\
  & & nasa93\_center\_2   & 	37  \\
  & & nasa93\_center\_5   & 	39  \\\hline	
  desharnais & language type & desharnaisL1   & 	46  \\
  & & desharnaisL2   & 	25  \\
  & & desharnaisL3   & 	10  \\\hline
finnish & application type & finnishAppType1 & 17 \\
 &  & finnishAppType2345 & 18 \\\hline
kemerer & hardware & kemererHardware1 & 7 \\
 &  & kemererHardware23456 & 8 \\\hline
maxwell & application type & maxwellAppType1 & 10 \\
 &  & maxwellAppType2 & 29 \\
 &  & maxwellAppType3 & 18 \\\hline
maxwell & hardware & maxwellHardware2 & 37 \\
 &  & maxwellHardware3 & 16 \\
 &  & maxwellHardware5 & 7 \\\hline
maxwell & source & maxwellSource1 & 8 \\
 &  & maxwellSource2 & 54 \\
\end{tabular}\caption{
6 datasets are selected from 20+ candidates.
Then selected datasets are divided into subsets according to a criterion that can define a \textit{cross-within division}. The datasets, subset sizes as well as the selection criteria are provided here.}
\label{fig:cc-selection}
\end{figure}

%\newpage
\subsection{Instance Selection and Retrieval}
The goal of our experiment is to determine at what probability
a learner retrieves
training instances from
either cross- or within- sources. The learner used in this
study is 
TEAK~\cite{Kocaguneli2011}, which is best explained as an 
extension to ABE0~\cite{Kocaguneli2010,Kocaguneli2011}.

\subsubsection{ABE0}

Analogy-based estimators
(ABE) generate an estimate for a \textit{test} project by retrieving similar past projects (a.k.a. analogies) from a database of past projects and adapting their effort values into an estimate.
We use ABE methods in this study since 1) they are 
widely investigated methods in the literature~\cite{Mendes2003, Chang1974, Keung2008a, Li2009, Kadoda2000,Kocaguneli2010,Kocaguneli2011}, 2) they are particularly helpful for \textit{cross} source
studies as they are based on distances between individual project instances.

There are various design options associated with ABE methods such as the distance measure for nearness~\cite{Mendes2003}, adaptation of analogy effort values~\cite{Mendes2003}, row processing~\cite{Chang1974, Keung2008a}, column processing~\cite{Keung2008a, Li2009} and so on.
Elsewhere we show that these options can easily lead to more than 6000 ABE variants~\cite{Keung2011}. 
Here we define ABE0 that is a {\em baseline} ABE method that combines
the tools used in Kadoda \&
Shepperd~\cite{Kadoda2000},  Mendes et al.~\cite{Mendes2003}, and Li et al.~\cite{Li2009}:
\bi
\item Input a database of past projects
%, whose rows are project instances and whose columns are \textit{independent} variables (that define projects) and a \textit{dependent} variable (effort value). 
%\item Decide how many similar projects (\textit{analogies}) are to be used from the training set, i.e \textit{k}-value.
\item For each test instance, retrieve \textit{k} similar projects (analogies).
\bi
\item For choosing \textit{k} analogies use a similarity measure.
\item Before calculating similarity, scale independent features to equalize their influence on the similarity measure.
\item Use a feature weighting scheme to reduce the effect of less informative features.
\ei
\item Adapt the effort values of the \textit{k} nearest analogies to come up with the effort estimate.
\ei
ABE0 uses the Euclidean distance as a similarity measure, whose formula is given in Equation \ref{equ:euclid}, where $w_i$ corresponds to feature weights applied on independent features.
ABE0 framework does not favor any features over the others, therefore each feature has equal importance in ABE0, i.e. $w_i=1$.
For adaptation ABE0 takes the median of retrieved \textit{k} projects.

\begin{equation}\label{equ:euclid}
Distance = \sqrt{\sum_{i=1}^{n}{w_i(x_{i} - y_{i})^{2}}}
\end{equation}

\subsubsection{TEAK}

TEAK is a {\em variance-based} instance selector that discards training data
associated with regions of high estimation variance.
It augments ABE0 with instance selection and an indexing scheme for filtering
relevant training examples.
Detailed description of TEAK can be found in~\cite{Kocaguneli2011}.
In summary, TEAK is a two-pass system:
\bi
\item Pass 1 prunes training instances implicated in poor decisions (instance selection);
\item Pass 2 retrieves closest instances  to the test instance (instance retrieval).
\ei


\begin{figure}[!b]
\begin{center}
\includegraphics[width=0.4\textwidth]{gac-tree-2.png}
\end{center}
\caption{A sample GAC tree with regions of high variance (red) and low variance (green). GAC trees may not always be binary. For example here, leaves are odd numbered, hence node ``c'' is left \textit{behind}. Such instances are pushed \textit{forward} into the closest node in the higher level. For example, ``c'' is pushed forward into the ``b+f'' node to make ``b+f+c'' node.  
}\label{fig:gac-tree}
\end{figure}


\begin{figure}[!b]
\begin{center}
~~~~~~~~GAC1~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~GAC2

\includegraphics[width=0.4\textwidth]{teak-2.png}
\end{center}
\caption{Execution of TEAK on 2 GAC trees, where tree on the left is GAC1 and the one on the left is GAC2 (i.e. lower variance sub-tree of GAC1). The instances in the low variance region of GAC1 (green region) are selected to form GAC2. Then test instance traverses GAC2 until no decrease in effort variance is possible. Wherever the test instance stops is retrieved as the subtree to be used for adaptation (white region of GAC2).   
}\label{fig:teak}
\end{figure}

In the first pass, training instances are combined using greedy-agglomerative clustering (GAC), to form an initial cluster tree that we call  GAC1; e.g.
\fig{gac-tree}.
Level zero of GAC1 is formed by leaves, which are the individual project instances.
These instances are greedily combined into tuples to form the nodes of upper levels.
The variance of the effort values associated with each sub-tree (the performance variance)
is then recorded and normalized $min .. max$ to $0 .. 1$.
The high variance sub-trees are then pruned, as these are the sub-trees that would 
cause an ABE method to make an estimate from a highly variable instance space. 
Hence, pass one prunes sub-trees with a variance greater than $\alpha\%$
of the maximum variance seen in any tree.  After some experimentation,
we found that $\alpha=10$ lead to estimates with lowest errors.

The leaves of the remaining sub-trees are the {\em survivors} of pass one.
They are filtered to pass 2 where they are used to build a second GAC tree (GAC2).
GAC2 is generated and traversed in a similar fashion to GAC1, then test instances are moved from root to leaves.
Unlike GAC1, this time variance is a decision criterion for the movement of test instances: If the variance of the current tree is larger than its sub-trees, then continue to move down; otherwise, stop and retrieve the instances in the current tree as the analogies.  
TEAK is a form of ABE0, so its adaptation method is the same, i.e. take the median of the analogy effort values.
A simple
visualization of this approach is given in in
\fig{teak}.

\begin{figure}[!t]
\renewcommand{\baselinestretch}{.5}
\centering
\small
  \begin{tabular}{lccccccccc}
& & & &\multicolumn{5}{c}{~~~~~~~~~~Simple ABE0}\\\cline{6-10}
&	\begin{sideways}\parbox{9mm}{TEAK}\end{sideways}	&	\begin{sideways}\parbox{9mm}{LR}\end{sideways}	&	\begin{sideways}\parbox{9mm}{NNet}\end{sideways}	&	\begin{sideways}\parbox{9mm}{\textit{k=best}}\end{sideways}	&	\begin{sideways}\parbox{9mm}{k=1}\end{sideways}	&	\begin{sideways}\parbox{9mm}{k=16}\end{sideways}	&	\begin{sideways}\parbox{9mm}{k=2}\end{sideways}	&	\begin{sideways}\parbox{9mm}{k=4}\end{sideways}	&	\begin{sideways}\parbox{9mm}{k=8}\end{sideways}	\\\hline
\multicolumn{10}{l}{  \underline{{\bf MdMRE}}}	\\
Cocomo81	   &	$\blacktriangle$ 	&		&		&	&		&		&		&		&		\\
Cocomo81e	    &	$\blacktriangle$ 	&		&		&		&		&		&		&		&		\\
Cocomo81o	    &	$\blacktriangle$ 	&		&		&		&		&		&		&		&		\\
Nasa93	        &	    &$\blacktriangle$ 	&	&		&		&		&		&		&		\\
Nasa93c2	    &	     &$\blacktriangle$	&	&		&		&		&		&		&		\\
Nasa93c5     	&	$\blacktriangle$ 	 &		&	&		&		&		&		&		&		\\
Desharnais	    &		     &$\blacktriangle$	&		&		&		&		&		&		&		\\
Sdr	            &	$\blacktriangle$	  &		&		&		& 	&		&		&		&		\\
ISBSG-Banking	&	$\blacktriangle$ 	 &		&		&		&		&		&		&		&		\\
\em Count	&	6	&	3	 &	0	&	0	&	0	&	0	&	0	&	0	&	0	\\


\hline\multicolumn{10}{l}{  \underline{{\bf Pred(25)}}}	\\
Cocomo81	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Cocomo81e	&		&		&	$\blacktriangle$	&		&		&		&		&		&		\\
Cocomo81o	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Nasa93	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Nasa93c2	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Nasa93c5	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Desharnais	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Sdr	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
ISBSG-Banking	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
\em Count	&	5	&	3	&	1	&	0	&	0	&	0	&	0	&	0	&	0	\\


\hline\multicolumn{10}{l}{ \underline{{\bf MAR}}}	\\
Cocomo81	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Cocomo81e	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Cocomo81o	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Nasa93	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Nasa93c2	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Nasa93c5	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Desharnais	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Sdr	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
ISBSG-Banking	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
\em Count	&	6	&	3	&	0	&	0	&	0	&	0	&	0	&	0	&	0	\\
\hline
\end{tabular}
\caption{
Results from
20 repeats
of a leave-one-out experiment,
repeated for the performance measures of MdMRE, Pred(25) and MAR.
Black triangles mark when an estimator was one of the top-ranked methods
for a particular data set (where ranking was computed via $win-loss$ from a Mann-Whitney test, 95\% confidence). 
The $Count$ rows show the number of times a method appeared as the top performing variant.
Results from~\cite{Kocaguneli2011}.
}
\label{fig:abe-vs-non-abe}
\end{figure}


\begin{figure}[!th]
\renewcommand{\baselinestretch}{1}
\begin{algorithmic} \footnotesize
\STATE \textit{$win_{i}$} = 0, \textit{$tie_{i}$} = 0, \textit{$loss_{i}$} = 0
\STATE \textit{$win_{j}$} = 0, \textit{$tie_{j}$} = 0, \textit{$loss_{j}$} = 0
\IF{Mann-Whitney(\textit{$P_i$}, \textit{$P_j$}) says they are the same}
	\STATE \textit{$tie_{i}$} = \textit{$tie_{i}$} + 1;
	\STATE \textit{$tie_{j}$} = \textit{$tie_{j}$} + 1;
\ELSE
	\IF{mean or median(\textit{$P_{i}$}) $<$ median(\textit{$P_{j}$})}
		\STATE \textit{$win_{i}$} = \textit{$win_{i}$} + 1
		\STATE \textit{$loss_{j}$} = \textit{$loss_{j}$} + 1
	\ELSE
		\STATE \textit{$win_{j}$} = \textit{$win_{j}$} + 1
		\STATE \textit{$loss_{i}$} = \textit{$loss_{i}$} + 1
	\ENDIF
\ENDIF
\end{algorithmic}
\caption{Pseudocode for win-tie-loss calculation between methods \textit{i} and \textit{j}
w.r.t. performance measures $P_i$ and $P_j$. Note here that only for Pred(30) the comparison is based on actual values ($Pred(30)_i$, $Pred(30)_j$) rather than mean or median values of performance measure arrays (median(\textit{$P_{i}$}), median(\textit{$P_{j}$})).}
\label{fig:win-tie-loss-pseudo}
\end{figure}


We use TEAK in this study since, as shown by the leave-one-out
experiments
of 
~\cite{Kocaguneli2011},
its performance is comparable 
with other commonly-used effort estimators
including neural networks (NNet) and linear regression (LR).
A summary of those performance results are 
given in \fig{abe-vs-non-abe} (for a complete analysis and for definitions of datasets please refer to Figure 7 of~\cite{Kocaguneli2011}).
That figure shows the results of a statistical comparison of various performance indicators (defined in~\cite{Keung2011})
for nine effort estimators and nine data sets from http://promisedata.org/?cat=14:
\bi
\item The columns $k=1,2,4,8,16$ denote variants of standard ABE0  where estimates are generated from
the $k$-th nearest neighbors.
\item The column $k=best$ denote a variant of ABE0 where $k$ was chosen by an initial pre-processor
that chose a best $k$ value after exploring the training data.
\item The columns $LR$ and $NNet$ refer to linear regression and neural nets.
\ei
The black triangles in \fig{abe-vs-non-abe} mark when an estimator was one of the top-ranked methods for a particular data set. 
Ranking was accomplished via the $win-loss$ calculation of \fig{win-tie-loss-pseudo}.
We first check if  two distributions $i,j$ are statistically different according to the Mann-Whitney test.
In our experimental setting, $i,j$ are arrays of performance measure results coming from two different methods.
If they are not statistically different, then they are said to \textit{tie} and we increment $tie_{i}$ and $tie_{j}$.  
On the contrary, if they are different, we updated $win_{i},win_{j}$ 
and ${loss_{i},loss_j}$ after a numerical comparison of performance measures.
The related pseudocode is given in \fig{win-tie-loss-pseudo}.
To get rid of any bias due to a particular experimental setting, for every experiment $20$ runs are made.
\begin{figure*}[!b]
\renewcommand{\baselinestretch}{1} 
\scriptsize
\centering
\begin{tabular}{l | c c c | c c | c c c | c c | c c c | c c | c c c | c c }
\textbf{Dataset} & \multicolumn{5}{c |}{\textbf{MAR}} & \multicolumn{5}{c |}{\textbf{MMRE}} & \multicolumn{5}{c |}{\textbf{MdMRE}} & \multicolumn{5}{c}{\textbf{Pred(30)}}  \\\hline
 &   &   & & \multicolumn{2}{c |}{\textbf{Median Results}} & & &  & \multicolumn{2}{c |}{\textbf{Median Results}} &  &  &   & \multicolumn{2}{c |}{\textbf{Median Results}} &  &  &   & \multicolumn{2}{c}{\textbf{Median Results}} \\
 & \textbf{W} & \textbf{T} & \textbf{L}  & \textbf{within}  & \textbf{cross} & \textbf{W} & \textbf{T} & \textbf{L}  & \textbf{within}  & \textbf{cross} & \textbf{W} & \textbf{T} & \textbf{L}  & \textbf{within}  & \textbf{cross} & \textbf{W} & \textbf{T} & \textbf{L}  & \textbf{within}  & \textbf{cross} \\
\input{winTieLossFileAll}
\end{tabular}
\caption{Results of TEAK: Comparison of performance between \textit{within} and \textit{cross} data w.r.t. 4 different performance measures (median of MAR, MMRE, MdMRE, Pred(30) over 20 runs) as well as \textbf{\textit{W, T, L}} statistics. Highlighted rows are the cases, where \textit{within} data is \textit{``dominantly''} better than \textit{cross}, i.e. wins more than half the time. Under the columns of \textbf{\textit{within}} and \textbf{\textit{cross}} the actual performance values associated with \textit{within} and \textit{cross} source
 datasets are provided respectively.}
\label{fig:cc-wc-performance-teak}
\end{figure*}
The key feature of \fig{abe-vs-non-abe} is that
TEAK always performed better than the other ABE0 methods, and usually  performed better than
 neural nets. TEAK's only
near-rival was linear regresison but, as shown in the $LR$ columns, TEAK was ranked top
nearly twice as much as linear regression.


\subsection{Experimentation}
The experimentation of this research has two different goals:
\bi
\item
The {\em   performance comparison}
of a state-of-the-art effort estimation method (TEAK)
when trained from \textit{within} and \textit{cross}
source data.
\item
 The {\em retrieval tendency}  goals question the
tendency of a \textit{within} test instance to retrieve \textit{within}
or \textit{cross} data.  In other words, given the chance that a
test instance had access to \textit{within} and \textit{cross} data
at the same time, what percentage of every subset would be retrieved
into \textit{k} analogies used for estimation?
\ei


\subsubsection{Performance Comparison}
For performance comparison we have two settings: \textit{Within}
and \textit{cross}.  In \textit{within} data setting, only
\textit{within} one source is used as the dataset and a testing strategy
of leave-one-out cross-validation (LOOCV) is employed.  LOOCV works
as follows: Given a \textit{within} dataset of $T$ projects, $1$
project at a time is selected as the test  and the remaining $T-1$
projects are used for  training, so eventually we have $T$ predictions.
The resulting $T$ predictions are then used to compute $4$ different
performance measures defined in \S2: PRED(30), MAR, MMRE, MdMRE. 

\textit{Cross} data setting uses \textit{within} data as the test
set and the \textit{cross} data as the training set.  In this setting
LOOCV is used as follows: each \textit{within} source is selected
as the test instance and TEAK derives an estimate for that
instance by adapting \textit{cross} analogies.  Ultimately we end
up with $T$ predictions adapted from a \textit{cross} dataset.
Finally the performances under \textit{within}
and \textit{cross} data settings are compared.  For that purpose
we use both mere performance values as well as win-tie-loss statistics.

\subsubsection{Retrieval Tendency}
For retrieval tendency, 
we select test instances according to LOOCV.
For each test instance, we are left with training sets of $T-1$ \textit{within} data and the subsets of \textit{cross} data.
After marking every \textit{within} and \textit{cross} instance, we combine the two datasets into a single training set and let the test instance choose analogies from the unified training set (note that analogies are retrieved after filtering in pass \#1 of TEAK).
In this setting our aim is to see what percentage of \textit{within} and \textit{cross} subsets would appear among retrieved \textit{k} analogies.
The retrieval percentage for a subset is the ratio of instances retrieved in analogies from that subset to its total size (see \eq{percentage}).

\begin{equation}\label{eq:percentage}
Percentage = \frac{SubsetSizeInAnalogies}{Subset Size}
\end{equation}

\newpage
\section{Results}
\label{sec:results}

\subsection{Performance Comparison}


For performance comparison 4 different performance measures are
employed: MAR, MMRE, MdMRE and Pred(30).  The actual performance
values are also evaluated subject to Mann Whitney
statistical test at $95\%$ confidence and this evaluation is
summarized by win-tie-loss statistics.  

\fig{cc-wc-performance-teak} shows \textit{within} and \textit{cross}
data performance when TEAK is used as the estimation method.  For
each performance measure win-tie-loss statistics  (abbreviated with
\textbf{W}, \textbf{T}, \textbf{L} respectively) of \textit{within}
performance when compared to \textit{cross} over 20 runs as well as
actual performance measure values are reported.  

The gray lines in 
\fig{cc-wc-performance-teak} show the experiments
where the $within$ results ``dominate'';
i.e.
win in more than half the comparisons.
Note that there are only  two gray lines.
In the remaining $\frac{19}{21}$ cases, the \textit{within} data
does not provide an advantage over \textit{cross} data.  In fact,
in one
case (kemererHardware1) the \textit{within} data is far
worse than \textit{cross} with an \textbf{L} value of 20.  These
results are confirmation of previous conclusions~\cite{Turhan2009,
Kocaguneli2010} in a much larger scale with $4$ error measures and
$21$ different cases: instance selection on \textit{cross} sources
improves its performance to an extent where it is no worse than
\textit{within} data.  


\begin{figure}[!b]
\renewcommand{\baselinestretch}{1}
\scriptsize
\centering
\begin{tabular}{l | l | l | l }
\textbf{Test Set} & \textbf{From S1} & \textbf{From S2}& \textbf{From S3} \\[1ex]\hline\hline
\input{resultsFile}
\end{tabular}
\caption{The average amount of analogies  (\textbf{\textit{k}}) retrieved from within and cross resource datasets by TEAK. In parenthesis the percentage of retrieved instances out of the actual \textit{within} source dataset is given. The diagonal entries that are highlighted with gray are the within source retrieval amounts and percentages.}
\label{fig:cc-selection}
\end{figure}


\subsection{Retrieval Tendency}

To explore  retrieval tendency,
LOOCV is used to choose single test instances one by one from a \textit{within} dataset of size $T$.
The remaining $T-1$ \textit{within} instances are combined with the \textit{cross} subsets.
Prior to combination, every training instance is marked with the
source that it belongs to (cross vs within).  Then the test instance
is allowed to choose \textit{k} analogies from a training set of
\textit{within} and \textit{cross} data.  

The rig lets us check the percentage retrieval of analogies from each one of the \textit{within} and \textit{cross} subsets.  
Those results are shown in \fig{cc-selection}.
Each \textit{cross-within division} is represented with a row of $2$ or
$3$ subsets; columns named \textit{``From $S_i$''} where $i\in\{1,2,3\}$
represent the subsets of the rows:
\bi
\item
  The highlighted diagonal entries
of each cell show the amount of instances
retrieved from \textit{within} subset.  
\item
The off-diagonal
values are the amount of instances retrieved from \textit{cross}
datasets.
\ei

To better see the percentages of \textit{within} and \textit{cross}
subsets, we sorted and plotted them in \fig{teak-perc}.  
\fig{teak-percentage} shows the sorted percentage values, where the $within$ percentages are shown with circles, whereas the $cross$ percentages are represented by triangles. 
Observe how the \textit{cross} percentage values are shifted versions of \textit{within} percentages (this shift-effect comes from the fact that there are more \textit{cross} subsets than \textit{within} subsets).  

The percentiles from  $10^{th}$ to
$90^{th}$ with increments of $20$ are given in \fig{teak-percentile}.
When we plot the percentiles, the shift-effect due to subset number
disappears and we are able to observe that within
and cross retrieval tendencies
at the indicated percentile values are very close.
A statistical test (Mann-Whitney, 95\% confidence)
confirms this: the distributions of 
\fig{teak-perc} are {\em
not} statistically significantly different.

\begin{figure}[!t]
  \centering                
  \subfigure[Percentages]{\label{fig:teak-percentage}\includegraphics[width=0.44\textwidth]{lib/teakWCCCPercentages.pdf}}
  \subfigure[Percentiles]{\label{fig:teak-percentile}\includegraphics[width=0.28\textwidth]{lib/teakWCCCPercentiles.pdf}}
  \caption{Percentages and percentiles of instances retrieved by TEAK from within and cross datasets. The cross percentages are very similar to shifted version of within percentages, the shift-effect is due to different number of subsets. The percentile graph removes the shift-effect and we see that \textit{within} test instances retrieve very close percentages of \textit{within} and \textit{cross} instances.}
  \label{fig:teak-perc}
\end{figure}


\section{Discussion}
\subsection{Implications}

We have shown above that,
for boundaries defined by a single feature:
\bi
\item
There was usually no difference in the  performance of effort estimators;
learned from {\em within} or from {\em across}
those
boundaries;
\item
There was usually no difference in the  probability of retrieving instances
for those estimates from  {\em within} or from {\em cross}
those
boundaries.
\ei
That is, it was not useful to 
divide the data by {\em any} of single feature boundaries shown in \fig{cc-selection}; i.e.
by project type, geographical location of the development center, language type,
application type, hardware, or source.
Hence,  at least for the purposes of selecting and retrieving relevant examples
for effort estimation, there is no information gain in dividing data using a single feature.

\subsection{Small Retrieval Sizes}
The median values of the percentiles
(i.e. $50^{th}$ percentile) in \fig{teak-perc}
is 7\%.
Initially, this low value troubled us but after a review
of the relevant literature we found that our
results are consistent with prior results:
\bi
\item
Chang's prototype generators~\cite{Chang1974} replaced training sets
of size \mbox{$T=(514,150,66)$} with prototypes of size \mbox{$N=(34,14,6)$}  (respectively).
\item
That is,  prototypes may be as few as $\frac{N}{T}=(7,9,9)\%$ of the original
data. Note that these values are close to how many instances were retrieved in the above
results.
\ei
\subsection{Geometric Implications}

\begin{wrapfigure}{r}{1in}
\includegraphics[width=1in]{xy}
\caption{Projects described in two dimensions.}\label{fig:dim}
\end{wrapfigure}
Our results imply something about the location of training data
in instance space.
Geometrically, $locality(1)$ assumes that project data
lines up along
one dimension; e.g. as shown by the circles in \fig{dim}.
This figure displays
projects described in terms of a two dimensional instance
space (labeled here as $x$ and $y$).  Note that (a) the circles are
arranged (approximately) parallel to the $y$ axis and that (b) the
longer projects (indicated with larger circles) occur at higher $y$
values. The space of the circle examples in \fig{dim} could be
processed by $locality(1)$ since a single feature (in this case, the
vertical $y$-axis) usefully divides the examples with higher effort
from those with lower effort.

From a geometic perspective,  $locality(1)$ is improbable.
Given the idiosyncrasies of software development, we find it
highly unlikely that naturally occurring project examples will all line
up in a row parallel to one axis. 
What is more likely, we believe, are geometrics like those shown as
squares in \fig{dim}. As before, the size
of the shapes  indicates the effort associated with each project.
Note these examples do not run parallel to any feature and the
longer and shorter projects are not easily separated by either axis.
The space of small squares and larger squares cannot be divided by any simplistic
$locality(1)$ assumption.


\subsection{Threats to Validity}
\label{sec:threats-to-validity}
{\em External validity} questions whether the results can be generalized outside
the specifications of a study\cite{Milic2004}.  
For the purpose of external validity, we use of 21 within-cross dataset pairs.
Among 10 studies investigated by Kitchenham et al. in~\cite{Kitchenham2007}, 9 of them used single within-cross dataset pairs, and 1 study used 6 pairs.
In terms of external validity, this report has higher validity than a standard \textit{within} vs. \textit{cross} data comparison effort estimation study.

Another consideration for external validity is the employed methods.
%Due to the nature of our research questions, particularly regarding selection tendency, we utilized ABE methods.
There are thousands of possible ABE variants and there is no way that this study covers them all.
There is obviously need for future research that repeats these experimentations with different ABE variants.
However, experiments reported here include a filtering based variant (TEAK) built on a base variant (ABE0) and run on 21 within-cross pairs.
Therefore, the extent of the experimentation in this research offers enough support for the claims that 1) \textit{cross} data performs no worse than \textit{within} data and 2) a \textit{within} test instance tends to retrieve equally from \textit{within} and \textit{cross} projects. 


{\em Construct validity} (i.e. face validity) asks if we are measuring
what we actually intended to measure~\cite{Robson2002}.
Previous studies have concerned themselves with the construct validity of 
different performance measures for effort estimation (e.g. \cite{Stensrud2002}).
So as not to bias our conclusions due to a limited number of measures, we used $4$ different performance measures aided with win-tie-loss statistics.
%Another threat to construct validity is the formation of our \textit{cross-within divisions}.
%The proposed division criteria for \textit{within} and \textit{cross} data may as well define different corporations in real world.
%Also the use of available public data promotes the reproducibility of the conclusions of a research.
%However, we acknowledge that using \textit{within} and \textit{cross} data coming from completely different organizations would be a better option.
%On the other hand, the issue with using such proprietary data is the fact that they are difficult to be shared with research community and therefore proposed results cannot be reproduced, validated or refuted. 

In terms of {\em internal validity} of our results, there is one dimension of experimental conditions not explored.
We are making use of LOOCV, whose a possible alternative would be N-Way cross-validation.
In N-Way cross-validation, data is randomly divided into $B$ bins and each  bin is tested on a model learned from the combination of other bins (typical values for $B$ are 3 or 10).
From a theoretical point of view, not controlling the stability of our results across different testing strategies is a threat to validity, as different testing strategies entails different bias and variance conditions~\cite{Hastie2008}.
Elsewhere~\cite{Kocaguneli2011a}, we show that there is very little difference in the bias and variance values generated for LOOCV and N-way cross-validation. 
Since two testing strategies have similar bias-variance characteristics for effort datasets, we opted for LOOCV due to the fact that LOOCV is a deterministic procedure that can be exactly repeated by any other researcher with access to a particular data set.  
N-way cross-validation on the other hand requires a random number generator and a stratification heuristic (to maintain same class distribution in each bin).
Without access to exact same random number generator and stratification heuristic, it would be difficult for a researcher ``A'' to reproduce results of researcher ``B''.

\section{Conclusion}
We have shown that
when
using a state-of-the-art effort estimator (TEAC),
then after instance selection:
\be
\item
The $cross$ performance results are no worse than $within$
(see \fig{cc-wc-performance-teak});
\item
The probability that the estimator retrieves a training instance
from $cross$ or $within$ is the same (see \fig{teak-perc}.b).
\ee


Result \#1 grants us permission to compare cross-vs-within results
(since there is no performance delta between them).
Result \#2 shows that the single-feature divisions have no
bearing on effort estimation.
Coupled with the results of~\cite{Keung2011},
these results are strong support for $locality(N)$ since we have
confirmed both {\bf PREDICTION~1} and {\bf PREDICTION~2}.

This means that (to  repeat a comment made in our introduction),
the most similar software to what you are writing now may not be in the next office. 
Rather, it may be in an office on the other side of the world.
As shown here,
using instance selection tools like TEAK, it is possible to automatically 
find that relevant training data.


%As to our research questions:
%
%\textbf{RQ1: What can be said about \textit{within} and \textit{cross}
%data performances?} The results of TEAK experiments confirm the
%findings of prior work~\cite{Turhan2009, Kocaguneli2010} on a larger
%scale: After relevancy filtering \textit{cross} data performance
%is comparable to that of \textit{within} data. 
% 
%\textbf{RQ2: What is the selection tendency of \textit{within} instances?}
%A surprising result of this research is the fact that under different
%ABE models (TEAK, ABE0, log+ABE0) test instances selected analogies
%from \textit{within} as well as \textit{cross} data.  This may be
%interpreted as a hint that organizations with very limited
%\textit{within} data can incorporate \textit{cross} data into their
%database.
%
%\textbf{RQ3: Which features are likely to define \textit{cross} and \textit{within} boundaries?}
%Our aim with this research question was to find features that would define boundaries to separate \textit{within} and \textit{cross} data from one another so that test instances would select mostly from its related \textit{within} dataset.
%However, none of the features we have identified resulted in such a scenario.
%On the contrary, they defined links between \textit{cross} datasets so that test instances selected from all subsets.

\section{Future Directions}
Some of the most likely future directions to this research are:
\bi
\item Reproduction of this work on proprietary data.
\item Investigating why particular subsets (cocomo81s, desharnaisL1) favor \textit{within} data, whereas the rest favors both \textit{within} and \textit{cross}.
\item Using different ABE or non-ABE methods under similar settings.
\item Experimenting if limited \textit{within} data can be supplemented with \textit{cross} data.
\item Using different features on different datasets to see if they can define a border between \textit{within} and \textit{cross} data.
\ei


\bibliographystyle{abbrv}
\bibliography{library}

\balancecolumns
% That's all folks!
\end{document}


While it was not their intention to do so,
a recent ESEM paper on algorithmic methods
by Mair and Shepperd~\cite{Mair05}
comments on  {\bf PREDICTION~1}.
That paper searched the literature
for consistent results regarding the value
of regression or analogy-based methods for software effort estimation. 
If Mair and Shepperd had found that analogy always out-performs regressions, 
then that would have
been strong support for {\bf PREDICTION~1} since 
(a) regression builds one model across all variables;
and (b)
analogy-based methods  
use the $k$-thh nearest neighbors discussed in the introduction; i.e. they make estimates
using the subset of training data that clumps nearest the test instance.
However, their results were otherwise:
from a total of 20 empirical
studies, seven favored regression, four were indifferent and nine favored analogy.

Note that Mair and Shepperd did not re-run experiments: rather, they commented
on the consistency of empirical results generated by other researchers.
Keuing et al.~\cite{Keung2011} lament the state of empirical methods
in effort estimation. It is well established that any experiment ordering $M$ effort estimators
will generate different rankings, depending on the accuracy statistic used in the 
comparison; the random number generators used to build training and test sets;
and the data sets used~\cite{shepperd2003}. Unless these factors are controlled,
we must expect inconsistent results such as those reported
by Mair and Shepperd.

In an attempt to improve the state of empirical methods, Keung et al. explored the size
of the 
