\documentclass{acm_proc_article-sp}

\usepackage{times}
\usepackage{cite}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{multirow}
\usepackage{rotating}
\usepackage{subfigure}


%\usepackage[none]{hyphenat} 
\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}%
  \setlength{\partopsep}{0pt}%
  \setlength{\parskip}{0pt}%
  \begin{itemize}%
  \setlength{\leftmargin}{.2in}%
  \setlength{\parsep}{0pt}%
  \setlength{\parskip}{0pt}%
  \setlength{\itemsep}{0pt}}%
 {\end{itemize}}

\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
   \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}


\usepackage[table]{xcolor}
\usepackage{url,graphicx}
\newcommand{\G}{\cellcolor[rgb]{0.8,0.8,0.8}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\hyp}[1]{Hypothesis~\ref{hyp:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}

\newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
\newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}

\begin{document}

\title{How to Find Relevant Data for Effort Estimation?} 


\numberofauthors{2} %  in this sample file, there are a *total*
% of EIGHT authors. SIX appear on the 'first-page' (for formatting
% reasons) and the remaining two appear in the \additionalauthors section.
%
\author{
% 1st. author
\alignauthor
Ekrem Kocaguneli\\
%	   \titlenote{Dr.~Trovato insisted his name be first.}\\
       \affaddr{Lane Department of Computer Science and Electrical Engineering}\\
       \affaddr{West Virginia University}\\
       \affaddr{Morgantown, USA}\\
       \email{ekocagun@mix.wvu.edu}
% 2nd. author
\alignauthor
Tim Menzies\\
       \affaddr{Lane Department of Computer Science and Electrical Engineering}\\
       \affaddr{West Virginia University}\\
       \affaddr{Morgantown, USA}\\
       \email{tim@menzies.us}
}
%\date{30 July 1999}
% Just remember to make sure that the TOTAL number of authors
% is the number that will appear on the first page PLUS the
% number that will appear in the \additionalauthors section.

\maketitle
\begin{abstract}
~\\
\textit{\textbf{Background}}: Building effort estimators requires
the correct selection of training data.  How to find that data?
Generating a database of past projects \textit{within} an organization
requires considerable investment of 1) time, 2) money and 3) educated
personnel.  Hence, it is tempting to \textit{cross} the boundaries
of development type, location, language, application and hardware
to use existing datasets of other organizations.
\\
\textit{\textbf{Aim}:} Prior results are inconclusive about the
merits of cross-resource data.  We conjecture that {\em instance
selection} can automatically find the instances most relevant, from
either within or acrcoss source dividisions.
\\
\textit{\textbf{Method}:} We filtered out 8 \textit{cross-within
divisions} (21 pairs of \textit{within-cross} subsets) out of 19
datasets and evaluated these divisions under different analogy-based
estimation (ABE) methods.
\\
\textit{\textbf{Results}:} The probablity of selecting training
data from within or across divisions based on single features is
the same.
\\
\textit{\textbf{Conclusion}:}
\end{abstract}


\category{H.4}{Software Cost Estimation}{\textit{k}-NN}
\category{D.2.8}{Software Engineering}{Cost}[within resource, cross resource]

\newtheorem{fore}{{\bf PREDICTION}}


\section{Introduction}

How is it possible that we can predict software development effort
with any degree of accuracy (e.g. 
within 30\% of actuals, 70\% of the time~\cite{chulani99})?
The space of possible projects is vast, especially compared
to the small number of examples found in effort estimation data sets. 
For example, the effort estimation datasets used in
Mendes et al.~\cite{Mendes2003}, Auer et al.~\cite{Auer2006},
Baker~\cite{baker07}, this study, and Li et al.~\cite{Li2009} have
median sizes of (13,15,31,33,52), respectively. We find it higly unlikely
that so few examples can cover the trends seen in modern software engineering-
especially if those examples were 
evenly spread across the space
of possible projects.

Clearly, effort estimation succeeds since
projects {\em clump together} in tiny regions of the total space.
Estimation works by reasoning about similarities found in those tiny clumps.
This paper asks the question {\em what is the best way to find those clumps}?




One method, supported by this paper, we shall call  ``{\em locality(N)}'' 
assumes that
effort estimation data  forms a  complex multi-dimensional space that can only
be  usefully divided {\em using multiple features}.
If $locality(N)$ is true, then it  would be {\em useless} to divide data by:
\bi
\item
Just the {\em project type} being developed (e.g. embedded, organic, etc);
\item
Or just the {\em development centers}  of the different developers;
\item
Or just the {\em development language} of the projects;
\item
Or just the {\em application type} (MIS\footnote{MIS=management information system}, 
GNC\footnote{GNC= guidance, navigation, and control}, etc);
\item
Or just the {\em targeted hardware platform};
\item
Or just the {\em in-house or outsourced} development;
\item
Or any other separation {\em based on a single feature}.  
\ei
In the
field of effort estimation, $locality(N)$ is {\em not} a common
assumption.  Many
publications~\cite{stukes98,stukes91,stukes99,ferens98,habib98,lum02,boehm00a}
including several of our own~\cite{menzies05a,Menzies2006} either
explicatively or tacitly assume ``$locality(1)$''; i.e.  clumps of
similar projects can be discovered {\em using a single  feature}.  For
example, some authors claim that, for a project in a specific
organization, software effort models work better when calibrated with
local data collected within that same organization. Proponents of such
a {\em within source} approach assume that it is best to divide
the training data according to the value of a single feature called
something like ``{\em companyName}''.
 


Confirming (or refuting) $locality(1)$
is an open and urgent question for effort estimation.
If $locality(1)$ is true, then any lessons learned
from one organization may never apply to another.
If so,
then our ability to
make general conclusions about software engineering would
be confined to small,
highly specialized,
sub-groups (e.g. just one company).

If, on the other hand, $locality(N)$ were true, then 
that would mean that
relevant data for effort estimation does not come just from
small sub-groups within one organization. Rather,  useful data could be collected
from many projects including \textit{cross sources}. 
This would simplify the construction of effort models for new projects:
just search other contexts for the right data 
for the new project.
Also, it could lead to conclusions about software engineering
that are general to many  development contexts.

We begin by making two predictions that would support $locality(N)$ and
would contradict $locality(1)$:
\begin{quote}{\em
\begin{fore}
Effort models built from training data divided on a single  feature
will perform no better (and, perhaps, even worse) than those that divide the data
using multiple features.
\end{fore}
\begin{fore}
Consider project data that has  been grouped into divisions
according to the value of a feature.
If training data is selected from within and cross those divisions,
then 
it would be $equally$ as probable to find useful data {\em within} as {\em cross} those
divisions.
\end{fore}
}
\end{quote}
Recent research offers much support for
{\bf PREDICTION~1}.
In a study
of 90 effort estimation methods (ten pre-processors $\times$  nine
learners),
we found that the best methods
were those that divided the training data according to multiple features~\cite{keung11}.
That result is detailed in our {\em Related Work}
section.

Turning now to {\bf PREDICTION~2},
 this paper tests if  divisions
based on {\em single} features improve effort estimation.  
We will allow an {\em instance selection} algorithm to find which instances are best
for training. That algorithm will be given access to all the training data, or
just the data divided via a single feature.
It  will be shown that:
\begin{quote} {\em 
The probability of selecting
training data from \textit{within} or \textit{cross} divisions based on single features \underline{is the same}.}
\end{quote}
This result, plus the results of Keung et al.~\cite{keung11}, are strong support for $locality(N)$
since they confirm both {\bf PREDICTION~1} and {\bf PREDICTION~2}.
These results have numerous practical implications:
\bi
\item Effort estimation is not dependent on some artificial, and useless,
 division of training data such as 
the source organization; the kind of application; or any other single feature
division of the data. 
Given the complex multi-dimensional nature of the software creation process, 
divisions of training data based on (say) geographical dimensions may be less important than other factors. 
The most similar software to what you are writing now may not be in the next office. 
Rather, it may be in an office on the other side of the world 
(and automatic {\em instance
selection algorithms} can  find that data).
\item It is useful to build effort estimation models from data taken from multiple contexts.
Hence:
\bi
\item Building estimation models is less expensive
since generating local models need not wait for an elaborate
(and expensive and time-consuming) collection process from local data.
\item
There are effects in software engineering that transcend our current divisions
of data (e.g. data from company1 or company2). This is a very exciting result since it
promises future generalizable SE results that apply to many organizations.
\ei
\item It offers a strong business case for collecting SE data in some sharable repository.
Such repositories can now be trusted to provide, at least some, relevant historical examples
for building effort estimators for some new project.
\ei
Before continuing, we 
offer one terminology clarification.
Other papers such as
Kitchenham  et al.~\cite{Kitchenham2007} and
Zimmermann et al.~\cite{Zimmermann2009} 
discuss the impact of dividing data according to a single
feature (either 
the {\em organization} or {\em application} name).
This paper explores those single-feature division {\em as well as} other single feature divisions
such as the {\em project type} being developed; the {\em development centers}
of the developers within
one organization; the \textit{development language} of the software; the  {\em application type}; the {\em targeted hardware platform}; and \textit{in-house or outsourced} development. 
So whereas (e.g.) Kitchenham et al. explore cross-vs-within company data sources,
we explore cross-vs-within data ``{\em sources}''  where the data is divided by the values
of \textit{any} single feature.
As shown below,
we find no examples where single feature division improves a state-of-the-art
effort estimation method.

\section{Related Work}
\subsection{Evidence for PREDICTION 1}

Keung et al.~\cite{keung11} built 90 effort estimators using
10 pre-processors and
9 learners.
Pre-processors included normalization, various discretization methods and
feature selection (PCA, stepwise, sequential forward). 
Learners included  $k=1$ and $k=5$-nearest-neighbor, linear and stepwise regression, 
CART, neural nets and PCR.

The 90 estimators were assessed via multiple accuracy statistics.
Let  $t$ instances have actuals $a_1,a_2,..,a_t$.
Prediction models generate prediction $p_i$ for  instance $i$.
If $|p_i - a_i|$ is $AR_i$ (the  {\em absolute residual}  difference between  predictions and actuals)
then:

\bi
\item
$MAR$ is the {\em mean absolute residual} $(\sum_i p_i - a_i)/t$.
\item
$MRE_i$ and $MER_i$ are the ratios  
$AR_i/a_i$ and 
$AR_i/p_i$.
\item
$PRED(25)$ is the percent of $t$ with $MRE_i \le 25\%$.
\item
$MMRE =\sum_i MRE_i/t$
\item
$MMER = \sum_i MER_i/t$.
\item
$MdMRE$ is median $MRE_i$ value. 
\item
Finally, the {\em balanced errors} are\newline
\mbox{$MBRE_i = (p_i - a_i )/ min(p_i,a_i)$} and\newline 
\mbox{$MIBRE_i = (p_i - a_i )/ max(p_i,a_i)$}
\ei

The 90 estimators were executed in a leave-one-out study
on  twenty data sets from http://promisedata.org/?cat=14
and their results compared using a Mann-Whitney test (95\% confidence).
In a result that might have been predicted by Shepperd et
al.~\cite{Shepperd2002}, the ranking of many of the estimators varied
across the different data sets and the different accuracy estimators.

However, Keung et al. found a small group of 13 estimators that were
consistently the best performers across all data sets (measured
according to all of MAR, MRE, PRED(25), MMRE, MMER, MdMRE, MBRE, and
MIBRE).  
In terms of this paper, 
the major result of Keung et al. is that all these 13 estimators
used CART or $k=1$ nearest neighbor. This is significant since
both these estimators use multiple features to sub-divide the training data:
\bi
\item
$k$-th nearest neighbor algorithms use all project features (perhaps, weighted by some
feature) to determine related projects~\cite{shepperd97};
\item
Tree-based algorithms like CART~\cite{Breiman1984} divide data into multiple
branches,  where each branch tests and divides that data on
multiple features.
\ei
Hence, this result is  strong support for {\bf PREDICTION~1}.


\subsection{Other Evidence}

Other results in the literature are also inconclusive about  $locality(N)$.
In their review of papers building effort models using data from within one
company or across
multiple companies, Kitchenham et al.~\cite{Kitchenham2007} found  equal evidence for and against the value of
building effort models based on a single feature division (specifically, they
found four studies favoring the use of \textit{within} company
data, and another three reporting that  using
\textit{cross} data performance is not significantly worse than \textit{within}). 
In other work, in the field of defect prediction,
Zimmermann et al.~\cite{Zimmermann2009} found that predictors
performed worse when trained from cross-application data than from within-application data. The evidence for their conclusion is quite emphatic:
{\em within} defeated {\em cross} in
618 out of 622 comparisons.

On the other hand, support for $locality(N)$
comes from the work of 
of Turhan et al.~\cite{Turhan2009},
and 
Kocaguneli et al.~\cite{Kocaguneli2010}.
Turhan et al. compare defect predictors learned from cross or within resource data.
Like Zimmermann, they found that using {\em all} cross resource data lead to poor predictor
performance (very large false alarm rates). However, after {\em instance selection} pruned
away irrelevant cross resource data, they found that the cross resource predictors were
equivalent to the predictors learned from within resource data~\cite{Turhan2009}.
Inspired by Turhan et al.,
Kocaguneli et al.~\cite{Kocaguneli2010} used {\em instance selection}
as a pre-processor for a study on cross-vs-within resource effort estimation.
In a limited study with
three data sets, they found that after pruning away irrelevant instances, 
the performance differences in
the predictors learned from cross or within data were statistically insignificant.

\subsection{Resolving the Evidence}

The Turhan, Kocaguneli et al. results clearly support $locality(N)$ but the other
results discussed above are either inconsistent or unsupportive.
How are we to reconcile this conflicting evidence?
One way is to note that:
\bi
\item Studies supporting $locality(N)$ all used instance selection filtering.
\item Instance selection filtering is {\em not} seen in the
Kitchenham, Zimmermann et al. studies.\footnote{ekrem: I updated the text here! We use \textit{instance selection} as a pre-processor here. In that respect, \textit{NONE} of the papers use \textit{instance selection as a pre-processor}. However, a 7 out of 10 studies in Kitchenham Survey use analogy-based estimation. In fact the \cite{Kitchenham2007} is contradictory: Table3 reports one study using ABE, whereas Table5 reports 7 studies using ABE, so I went back and checked all studies and ABE employing papers are 7 out of 10!}
\ei
%The one exception is that 
%one study surveyed by  Kitchenham et al. explored  analogy-based estimation
%and, in that analogy-based result, training on cross resource data worked as well as 
%training on within resource data. 
An \textit{instance selection} filtering uses every feature (perhaps, with
some feature weighting) to find relevant training examples.
Hence, the studies with \textit{instance selection} filtering found in~\cite{Turhan2009, Kocaguneli2010}  offer more  support
for  $locality(N)$ than for $locality(1)$.
However, ~\cite{Turhan2009} and \cite{Kocaguneli2010}
are hardly conclusive since they do not collect the information required to
comment on {\bf PREDICTION~2}. What is required is a well-controlled
 instance selection
experiment over 
data divided by using some single feature. {\bf PREDICTION~2} would be supported
if the instance selector selects as much data {\em within} as {\em across}
the single feature divisions. 

\fig{problem-types} illustrates how this study differs to prior work.
Most effort estimation research falls into \fig{problem-types}.a where 
estimation models are applied {\em within} one source set to learn an estimator.
Examples of this approach include \cite{Mendes2003,Auer2006,Li2009,Shepperd1997,Menzies2006}.

A smaller number of papers, such as those surveyed by Kitchenham et al.~\cite{Kitchenham2007},
explore building models using data that falls {\em across} many sources (see \fig{problem-types}.b).
Fewer still are the papers like ~\cite{Turhan2009,Kocaguneli2010}).
that, prior to learning,
apply some instance selection filter to cross resource data
sources (see \fig{problem-types}.c).

To the best of our knowledge, this paper is the first that allows an effort estimator 
to select training data from  either {\em cross} or {\em within} different sources,
then checks what data was selected from
which source  (see \fig{problem-types}.d).

\begin{figure}[th!]
  \centering                
  \subfigure[\textit{Within}]{\label{fig:within-problem}\includegraphics[height=6cm, width=1.5cm]{withinProblem-2.png}}~~
  \subfigure[\textit{Cross}]{\label{fig:cross-problem}\includegraphics[height=6cm, width=1.5cm]{crossProblem-2.png}} ~~
  \subfigure[\textit{Filter}]{\label{fig:cross-filter-problem}\includegraphics[height=6cm, width=1.5cm]{crossFilterProblem-2.png}} ~~
  \subfigure[\textit{Within+Cross}]{\label{fig:within-cross-problem}\includegraphics[height=6cm, width=3cm]{withinCrossProblem-2.png}} 
  \caption{The problem types in \textit{within} vs. \textit{cross} data comparison and our conclusions so far. \textit{``+''} and \textit{``-''} signs on top of each approach
 mean positive and negative results respectively. A \textit{``?''} sign means that
the approach  has not previously been investigated.}
  \label{fig:problem-types}
\end{figure}

\section{Methodology}

The details of dataset filtering and division of them into \textit{cross-within divisions} are presented here.
We also explain the selected performance measures and our experimental set-up.

\subsection{Datasets}
There are 2 fundamental factors that were considered for selection of the datasets used in this research:
\bi
\item Public availability: For reproducibility purposes
\item Cross-within divisibility: For enabling \textit{cross} vs. \textit{within} experimentation
\ei
A critical issue in software engineering is the ability of the proposed results to be reproducible~\cite{Juristo2009, Kitchenham2007} and use of proprietary data is a major obstacle towards this goal.
Therefore, all our datasets are publicly available through PROMISE data repository~\cite{Boetticher2007}.

We define \textit{cross-within division} as the subset(s) of effort
data that are formed through division of one feature:
instances having the same value for that feature  form a
subset.  
Such features are plausible candidates for generating 
a \textit{cross} source experiment, i.e. the features should be
likely to change from one source to other.
Accordingly,
this study began by exploring what PROMISE effort data
can be divided via a single feature.
After manually inspecting more than 20 datasets,
six were found to be suitable for \textit{cross-within} experimentation.
Those six data sets support the
21
\textit{cross-within divisions} shown in \fig{cc-selection}.
The selected division criteria include:
\bi
\item project type: embedded, organic and semidetached (\textit{cocomo81}), 
\item center: geographical development center (\textit{nasa93}), 
\item language type: programming language used for development (\textit{desharnais}), 
\item application type: on-line service program, production control program etc. (\textit{finnish} and \textit{maxwell}), 
\item hardware: PC, mainframe, networked etc. (\textit{kemerer} and maxwell), 
\item source: whether in-house or outsourced (maxwell).
\ei
In the following, we will use the following nomenclatures.
If we name a data set, followed by a set of numbers,
these
correspond to values of the  feature used to form the subsets.
If a name has multiple numbers at the end (e.g. finnishAppType2345) this means that all instances with these  feature values are combined in a single subset.
%Also note that the terms \textit{within/cross} and WS/CS will be used interchangeably for the rest of the text.

\begin{figure}
\scriptsize
\center
\begin{tabular}{l | l | l | r}
\textbf{Dataset} & \textbf{Criterion} & \textbf{Subsets} & \textbf{Subsets Size}\\[1ex]\hline\hline
cocomo81 & project type & cocomo81e &	28  \\
  & & cocomo81o   & 	24  \\
  & & cocomo81s   & 	11	  \\\hline
  nasa93 & development center & nasa93\_center\_1   & 	12  \\
  & & nasa93\_center\_2   & 	37  \\
  & & nasa93\_center\_5   & 	39  \\\hline	
  desharnais & language type & desharnaisL1   & 	46  \\
  & & desharnaisL2   & 	25  \\
  & & desharnaisL3   & 	10  \\\hline
finnish & application type & finnishAppType1 & 17 \\
 &  & finnishAppType2345 & 18 \\\hline
kemerer & hardware & kemererHardware1 & 7 \\
 &  & kemererHardware23456 & 8 \\\hline
maxwell & application type & maxwellAppType1 & 10 \\
 &  & maxwellAppType2 & 29 \\
 &  & maxwellAppType3 & 18 \\\hline
maxwell & hardware & maxwellHardware2 & 37 \\
 &  & maxwellHardware3 & 16 \\
 &  & maxwellHardware5 & 7 \\\hline
maxwell & source & maxwellSource1 & 8 \\
 &  & maxwellSource2 & 54 \\
\end{tabular}\caption{
6 datasets are selected from 20+ candidates.
Then selected datasets are divided into subsets according to a criterion that can define a \textit{cross-within division}. The datasets, subset sizes as well as the selection criteria are provided here.}
\label{fig:cc-selection}
\end{figure}

\subsection{Instance Selection}
The goal of our experiment is to determine at what probability
an  agent selects
training instances from
either cross- or within- sources. The instance selector used in this
study is 
TEAK~\cite{Kocaguneli2011}, which is best explained as an 
extension to ABE0~\cite{Kocaguneli2010,Kocaguneli2011}.

\subsubsection{ABE0}

Analogy-based estimators
(ABE) generate an estimate for a \textit{test} project by retrieving similar past projects (a.k.a. analogies) from a database of past projects and adapting their effort values into an estimate.
We use ABE methods in this study since 1) they are 
widely investigated methods in the literature~\cite{Mendes2003, Chang1974, Keung2008a, Li2009, Kadoda2000,Kocaguneli2010,Kocaguneli2011}, 2) they are particularly helpful for \textit{cross} source
studies as they are based on distances between individual project instances.

There are various design options associated with ABE methods such as the distance measure for nearness~\cite{Mendes2003}, adaptation of analogy effort values~\cite{Mendes2003}, row processing~\cite{Chang1974, Keung2008a}, column processing~\cite{Keung2008a, Li2009} and so on.
Elsewhere~\cite{keung11} we show that these options can easily lead to more than 6000 ABE variants. 

ABE0 is a {\em baseline} analogy-based estimator which combines
the ABE tools used in Kadoda \&
Shepperd~\cite{Kadoda2000},  Mendes et al.~\cite{Mendes2003}, and Li et al.~\cite{Li2009}:
\bi
\item Input a database of past projects, whose rows are project instances and whose columns are \textit{independent} variables (that define projects) and a \textit{dependent} variable (effort value). 
\item Decide how many similar projects (\textit{analogies}) are to be used from the training set, i.e \textit{k}-value.
\item For each test instance, retrieve \textit{k} analogies from the database.
\bi
\item For selection of \textit{k} analogies use a similarity measure like Euclidean distance measure.
\item Before calculating similarity, scale independent features to equalize their influence on the similarity measure.
\item Use a feature weighting scheme to reduce the effect of less informative features.
\ei
\item Adapt the effort values of the \textit{k} nearest analogies to come up with the effort estimate.
\ei
ABE0 uses the Euclidean distance as a similarity measure, whose formula is given in Equation \ref{equ:euclid}, where $w_i$ corresponds to feature weights applied on independent features.
ABE0 framework does not favor any features over the others, therefore each feature has equal importance in ABE0, i.e. $w_i=1$.
For adaptation ABE0 takes the median of selected \textit{k} projects.

\begin{equation}\label{equ:euclid}
Distance = \sqrt{\sum_{i=1}^{n}{w_i(x_{i} - y_{i})^{2}}}
\end{equation}

\subsubsection{TEAK}

TEAK is a {\em variance-based} instance selector that discards training data
associated with regions of high estimation variance.
It augments ABE0 with instance selection and an indexing scheme for finding
relevant training examples.
Detailed description of TEAK can be found in~\cite{Kocaguneli2011}.

In summary, TEAK is a two-pass system:
\bi
\item Pass 1 prunes training instances implicated in poor decisions;
\item Pass 2 selects instances closest to the test instance.
\ei
In the first pass, training instances are combined using greedy-agglomerative clustering,
or GAC, to form an initial cluster tree we call  GAC1; e.g.
\fig{gac-tree}.
Level zero of GAC1 is formed by leaves, which are the individual project instances.
These instances are greedily combined into tuples to form the nodes of upper levels.
The variance of the effort values associated with each sub-tree (the performance variance)
is then recorded and normalized $min .. max$ to $0 .. 1$.
The high variance sub-trees are then pruned, as these are the sub-trees that would 
cause an ABE method to make an estimate from a highly variable instance space. 
Hence, pass one prunes sub-trees with a variance greater than $\alpha\%$
of the maximum variance seen in any tree.  After some experimentation,
we found that $\alpha=10$ lead to estimates with lowest errors.

The leaves of the remaining sub-trees are the {\em survivors} of pass one.
They are filtered to pass 2 where they are used to build a second GAC tree (GAC2).
GAC2 is generated and traversed in a similar fashion to GAC1, then test instances are moved from root to leaves.
Unlike GAC1, this time variance is a decision criterion for the movement of test instances: If the variance of the current tree is larger than its sub-trees, then continue to move down; otherwise, stop and select the instances in the current tree as the analogies.  
TEAK is a form of ABE0, so its adaptation method is the same, i.e. take the median of the analogy effort values.
A simple
visualization of this approach is given in in
\fig{teak}.


\begin{figure}[!t]
\begin{center}
\includegraphics[width=0.48\textwidth]{gac-tree-2.png}
\end{center}
\caption{A sample GAC tree with regions of high variance (red) and low variance (green). GAC trees may not always be binary. For example here, leaves are odd numbered, hence node ``c'' is left \textit{behind}. Such instances are pushed \textit{forward} into the closest node in the higher level. For example, ``c'' is pushed forward into the ``b+f'' node to make ``b+f+c'' node.  
}\label{fig:gac-tree}
\end{figure}


\begin{figure}[!t]
\begin{center}
\includegraphics[width=0.48\textwidth]{teak-2.png}
\end{center}
\caption{Execution of TEAK on 2 GAC trees, where tree on the left is GAC1 and the one on the left is GAC2 (i.e. lower variance sub-tree of GAC1). The instances in the low variance region of GAC1 (green region) are selected to form GAC2. Then test instance traverses GAC2 until no decrease in effort variance is possible. Wherever the test instance stops is selected as the subtree to be used for adaptation (white region of GAC2).   
}\label{fig:teak}
\end{figure}

\begin{figure}[!th]
\renewcommand{\baselinestretch}{.5}
\centering
\small
  \begin{tabular}{lccccccccc}
& & & &\multicolumn{5}{c}{~~~~~~~~~~Simple ABE0}\\\cline{6-10}
&	\begin{sideways}\parbox{9mm}{TEAK}\end{sideways}	&	\begin{sideways}\parbox{9mm}{LR}\end{sideways}	&	\begin{sideways}\parbox{9mm}{NNet}\end{sideways}	&	\begin{sideways}\parbox{9mm}{\textit{k=best}}\end{sideways}	&	\begin{sideways}\parbox{9mm}{k=1}\end{sideways}	&	\begin{sideways}\parbox{9mm}{k=16}\end{sideways}	&	\begin{sideways}\parbox{9mm}{k=2}\end{sideways}	&	\begin{sideways}\parbox{9mm}{k=4}\end{sideways}	&	\begin{sideways}\parbox{9mm}{k=8}\end{sideways}	\\\hline
\multicolumn{10}{l}{  \underline{{\bf MdMRE}}}	\\
Cocomo81	   &	$\blacktriangle$ 	&		&		&	&		&		&		&		&		\\
Cocomo81e	    &	$\blacktriangle$ 	&		&		&		&		&		&		&		&		\\
Cocomo81o	    &	$\blacktriangle$ 	&		&		&		&		&		&		&		&		\\
Nasa93	        &	    &$\blacktriangle$ 	&	&		&		&		&		&		&		\\
Nasa93c2	    &	     &$\blacktriangle$	&	&		&		&		&		&		&		\\
Nasa93c5     	&	$\blacktriangle$ 	 &		&	&		&		&		&		&		&		\\
Desharnais	    &		     &$\blacktriangle$	&		&		&		&		&		&		&		\\
Sdr	            &	$\blacktriangle$	  &		&		&		& 	&		&		&		&		\\
ISBSG-Banking	&	$\blacktriangle$ 	 &		&		&		&		&		&		&		&		\\
\em Count	&	6	&	3	 &	0	&	0	&	0	&	0	&	0	&	0	&	0	\\


\hline\multicolumn{10}{l}{  \underline{{\bf Pred(25)}}}	\\
Cocomo81	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Cocomo81e	&		&		&	$\blacktriangle$	&		&		&		&		&		&		\\
Cocomo81o	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Nasa93	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Nasa93c2	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Nasa93c5	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Desharnais	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Sdr	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
ISBSG-Banking	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
\em Count	&	5	&	3	&	1	&	0	&	0	&	0	&	0	&	0	&	0	\\


\hline\multicolumn{10}{l}{ \underline{{\bf MAR}}}	\\
Cocomo81	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Cocomo81e	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Cocomo81o	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Nasa93	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Nasa93c2	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Nasa93c5	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Desharnais	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Sdr	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
ISBSG-Banking	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
\em Count	&	6	&	3	&	0	&	0	&	0	&	0	&	0	&	0	&	0	\\
\hline
\end{tabular}
\caption{
Restuls fro
20 repeats
of a leave-one-out experiment,
repeated for the performance measures of MdMRE, Pred(25) and MAR.
Black triangles mark when an estimator was one of the top-ranked methods
for a particular data set (where ranking was computed via $win-loss$ from a Mann-Whitney test, 95\% confidence). 
The $Count$ rows show the number of times a method appeared as the top performing variant.
Results from~\cite{Kocaguneli2011}.
}
\label{fig:abe-vs-non-abe}
\end{figure}

\begin{figure*}[th!]
\renewcommand{\baselinestretch}{0.8} 
\scriptsize
\centering
\begin{tabular}{l | c c c | c c | c c c | c c | c c c | c c | c c c | c c }
\textbf{Dataset} & \multicolumn{5}{c |}{\textbf{MAR}} & \multicolumn{5}{c |}{\textbf{MMRE}} & \multicolumn{5}{c |}{\textbf{MdMRE}} & \multicolumn{5}{c}{\textbf{Pred(30)}}  \\\hline
 &   &   & & \multicolumn{2}{c |}{\textbf{Median Results}} & & &  & \multicolumn{2}{c |}{\textbf{Median Results}} &  &  &   & \multicolumn{2}{c |}{\textbf{Median Results}} &  &  &   & \multicolumn{2}{c}{\textbf{Median Results}} \\
 & \textbf{W} & \textbf{T} & \textbf{L}  & \textbf{within}  & \textbf{cross} & \textbf{W} & \textbf{T} & \textbf{L}  & \textbf{within}  & \textbf{cross} & \textbf{W} & \textbf{T} & \textbf{L}  & \textbf{within}  & \textbf{cross} & \textbf{W} & \textbf{T} & \textbf{L}  & \textbf{within}  & \textbf{cross} \\
\input{winTieLossFileAll}
\end{tabular}
\caption{Results of TEAK: Comparison of performance between \textit{within} and \textit{cross} data w.r.t. 4 different performance measures (median of MAR, MMRE, MdMRE, Pred(30) over 20 runs) as well as \textbf{\textit{W, T, L}} statistics. Highlighted rows are the cases, where \textit{within} data is \textit{``dominantly''} better than \textit{cross}, i.e. wins more than half the time. Under the columns of \textbf{\textit{within}} and \textbf{\textit{cross}} the actual performance values associated with \textit{within} and \textit{cross} source
 datasets are provided respectively.}
\label{fig:cc-wc-performance-teak}
\end{figure*}


We use TEAK in this study since, as shown by leave-one-out
experiments
of 
~\cite{Kocaguneli2011},
its performance is comparable 
with other commonly-used effort estimators
including neural networks (NNet) and linear regression (LR).
A summary of those performance results are 
given in \fig{abe-vs-non-abe} (for a complete analysis and for definitions of datasets please refer to Figure 7 of~\cite{Kocaguneli2011}).
That figure shows the results of a statistical comparison of various performance indicators (defined in~\cite{keung11})
for nine effort estimators and nine data sets from http://promisedata.org/?cat=14:
\bi
\item The columns $k=1,2,4,8,16$ denote variants of standard ABE0  where estimates are generated from
the $k$-th nearest neighbors.
\item The column $k=best$ denote a variant of ABE0 where $k$ was chosen by an initial pre-processor
that selected a best $k$ value after exploring the training data.
\item The columns $LR$ and $NNet$ refer to linear regression and neural nets.
\ei
The black triangles in \fig{abe-vs-non-abe} mark when an estimator was one of the top-ranked methods
for a particular data set. Ranking was accomplished via the $win-loss$ calculation of \fig{win-tie-loss-loss-pseudo}.
We first check if  two distributions $i,j$ are statistically different according to the Mann-Whitney test.
In our experimental setting, $i,j$ are arrays of performance measure results coming from two different methods.
If they are not statistically different, then they are said to \textit{tie} and we increment $tie_{i}$ and $tie_{j}$.  
On the contrary, if they are different, we updated $win_{i},win_{j}$ 
and ${loss_{i},loss_j}$ after a numerical comparison of performance measures.
The related pseudocode is given in \fig{win-tie-loss-pseudo}.
To get rid of any bias that would come from a particular experimental setting, for every experiment $20$ runs are made.

The key feature of \fig{abe-vs-non-abe} is that
TEAK always performed better than the other ABE0 methods, and usually  performed better than
 neural nets. TEAK's only
near-rival was linear regresison but, as shown in the $LR$ columns, TEAK was ranked top
nearly twice as much as linear regression.


\begin{figure}
\renewcommand{\baselinestretch}{1}
\begin{algorithmic} \footnotesize
\STATE \textit{$win_{i}$} = 0, \textit{$tie_{i}$} = 0, \textit{$loss_{i}$} = 0
\STATE \textit{$win_{j}$} = 0, \textit{$tie_{j}$} = 0, \textit{$loss_{j}$} = 0
\IF{Mann-Whitney(\textit{$P_i$}, \textit{$P_j$}) says they are the same}
	\STATE \textit{$tie_{i}$} = \textit{$tie_{i}$} + 1;
	\STATE \textit{$tie_{j}$} = \textit{$tie_{j}$} + 1;
\ELSE
	\IF{mean or median(\textit{$P_{i}$}) $<$ median(\textit{$P_{j}$})}
		\STATE \textit{$win_{i}$} = \textit{$win_{i}$} + 1
		\STATE \textit{$loss_{j}$} = \textit{$loss_{j}$} + 1
	\ELSE
		\STATE \textit{$win_{j}$} = \textit{$win_{j}$} + 1
		\STATE \textit{$loss_{i}$} = \textit{$loss_{i}$} + 1
	\ENDIF
\ENDIF
\end{algorithmic}
\caption{Pseudocode for win-tie-loss calculation between methods \textit{i} and \textit{j}
w.r.t. performance measures $P_i$ and $P_j$. Note here that only for Pred(30) the comparison is based on actual values ($Pred(30)_i$, $Pred(30)_j$) rather than mean or median values of performance measure arrays (median(\textit{$P_{i}$}), median(\textit{$P_{j}$})).}
\label{fig:win-tie-loss-pseudo}
\end{figure}

\subsection{Experimentation}
The experimentation of this research has two different goals:
\bi
\item
The {\em   performance comparison}
goal compares different ABE methods (TEAK and ABE0 with
\textit{k={4,best}}) when subject to \textit{within} and \textit{cross}
sourcey data.
\item
 The {\em selection tendency}  goals question the
tendency of a \textit{within} test instance to selection \textit{within}
or \textit{cross} data.  In other words, given the chance that a
test instance had access to \textit{within} and \textit{cross} data
at the same time, what percentage of every subset would be selected
into \textit{k} analogies used for estimation?
\ei


\subsubsection{Performance Comparison}
For performance comparison we have two settings: \textit{Within}
and \textit{cross}.  In \textit{within} data setting, only
\textit{within} one source is used as the dataset and a testing strategy
of leave-one-out cross-validation (LOOCV) is employed.  LOOCV works
as follows: Given a \textit{within} dataset of $T$ projects, $1$
project at a time is selected as the test  and the remaining $T-1$
projects are used for  training, so eventually we have $T$ predictions.
The resulting $T$ predictions are then used to compute $4$ different
performance measures defined in \S2: PRED(30), MAR, MMRE, MdMRE. 

\textit{Cross} data setting uses \textit{within} data as the test
set and the \textit{cross} data as the training set.  In this setting
LOOCV is used as follows: each \textit{within} source is selected
as the test instance and TEAK and ABE0 methods derive an estimate for that
instance by adapting \textit{cross} analogies.  Ultimately we end
up with $T$ predictions adapted from a \textit{cross} dataset.
Finally the performances of TEAK and ABE0 methods under \textit{within}
and \textit{cross} data settings are compared.  For that purpose
we use both mere performance values as well as win-tie-loss statistics.

\subsubsection{Selection Tendency}
For selection tendency, 
we select test instances according to LOOCV.
For each test instance, we are left with training sets of $T-1$ \textit{within} data and the subsets of \textit{cross} data.
After marking every \textit{within} and \textit{cross} instance, we combine the two datasets into a single training set and let the test instance choose analogies from the unified training set.
In this setting our aim is to see what percentage of \textit{within} and \textit{cross} subsets would appear among selected \textit{k} analogies.
The percentage for a subset is the ratio of instances selected as analogies from that subset to its total size (see \eq{percentage}).

\begin{equation}\label{eq:percentage}
Percentage = \frac{SubsetSizeInAnalogies}{Subset Size}
\end{equation}

\begin{figure}[!th]
\renewcommand{\baselinestretch}{0.8}
\scriptsize
\centering
\begin{tabular}{l | l | l | l | l }
\textbf{Test Set} & \textbf{\textit{k}} & \textbf{From S1} & \textbf{From S2}& \textbf{From S3} \\[1ex]\hline\hline
\input{resultsFile}
\end{tabular}
\caption{The average amount of analogies  (\textbf{\textit{k}}) selected from within and cross resource datasets by TEAK. In parenthesis the percentage of selected instances out of the actual within source dataset is given. The diagonal entries that are highlighted with gray are the within source selection amounts and percentages.}
\label{fig:cc-selection}
\end{figure}

\begin{figure*}[th!]
  \centering                
  \subfigure[Percentages]{\label{fig:teak-percentage}\includegraphics[width=0.5\textwidth]{lib/teakWCCCPercentages.pdf}}
  \subfigure[Percentiles]{\label{fig:teak-percentile}\includegraphics[width=0.28\textwidth]{lib/teakWCCCPercentiles.pdf}}
  \caption{Percentages and percentiles of instances selected by TEAK from within and cross datasets. The cross percentages are very similar to shifted version of within percentages, the shift-effect is due to different number of subsets. The percentile graph removes the shift-effect and we see that \textit{within} test instances select very close percentages of \textit{within} and \textit{cross} instances.}
  \label{fig:teak-perc}
\end{figure*}

\section{Results}
\label{sec:results}

\subsection{Performance Comparison}


For performance comparison 4 different performance measures are
employed: MAR, MMRE, MdMRE and Pred(30).  The actual performance
values are also evaluated subject to Mann Whitney
statistical test at $95\%$ confidence and this evaluation is
summarized by win-tie-loss statistics.  

\fig{cc-wc-performance-teak} shows \textit{within} and \textit{cross}
data performance when TEAK is used as the estimation method.  For
each performance measure win-tie-loss statistics  (abbreviated with
\textbf{W}, \textbf{T}, \textbf{L} respectively) of \textit{within}
performance when compared to \textit{cross} over 20 runs as well as
actual performance measure values are reported.  

The gray lines in 
\fig{cc-wc-performance-teak} show the experiments
where the $within$ results ``dominate'';
i.e.
win in more than half the comparisons.
Note that there are only  two gray lines.
In the remaining $\frac{19}{21}$ cases, the \textit{within} data
does not provide an advantage over \textit{cross} data.  In fact,
in one
case (kemererHardware1) the \textit{within} data is far
worse than \textit{cross} with an \textbf{L} value of 20.  These
results are confirmation of previous conclusions~\cite{Turhan2009,
Kocaguneli2010} in a much larger scale with $4$ error measures and
$21$ different cases: instance selection on \textit{cross} sources
improves its performance to an extent where it is no worse than
\textit{within} data.  


\subsection{Selection Tendency}

To explore  selection tendency,
LOOCV is used to select out single test instances one by one from a \textit{within} dataset of size $T$.
The remaining $T-1$ \textit{within} instances are combined with the \textit{cross} subsets.
Prior to combination, every training instance is marked with the
source that it belongs to (cross vs within).  Then the test instance
is allowed to choose \textit{k} analogies from a training set of
\textit{within} and \textit{cross} data.  

The rig lets us check
the percentage
selection of analogies from each one of the \textit{within} and
\textit{cross} subsets.  
Those results are shown in \fig{cc-selection}.
Each
\textit{cross-within division} is represented with a row of $2$ or
$3$ subsets; columns named \textit{``From $S_i$''} where $i\in\{1,2,3\}$
represent the subsets of the rows:
\bi
\item
  The highlighted diagonal entries
of each cell show the amount of instances
selected from \textit{within} subset.  
\item
The off-diagonal
values are the amount of instances selected from \textit{cross}
datasets.
\ei

To better see the percentages of \textit{within} and \textit{cross}
subsets, we sorted and plotted them
in
\fig{teak-perc}.  \fig{teak-percentage} shows the sorted percentage
values, where the $within$ percentages are shown with circles, whereas
the $cross$ percentages are represented by triangles. 
Observe how 
the \textit{cross} percentage values are
shifted versions of \textit{within} percentages  
(this shift-effect
comes from the fact that there are more \textit{cross} subsets than
\textit{within} subsets).  

The percentiles from  $10^{th}$ to
$90^{th}$ with increments of $20$ are given in \fig{teak-percentile}.
When we plot the percentiles, the shift-effect due to subset number
disappears and we are able to observe that within
and cross selection tendencies
at the indicated percentile values are very close.
A statistical test (Mann-Whitney, 95\% confidence)
confirms this: the distributions of 
\fig{teak-perc} are {\em
not} statistically significantly different.


\section{Discussion}
\subsection{Small Selection Sizes}
Initially, we were 
surprised about how $few$ instances
were selected 
by TEAK.
The median values of the percentiles
(i.e. $50^{th}$ percentile) in \fig{teak-perc}
is 7\%.
Initially, this low value troubled us but after a review
of the instance selection literature we found that this
was not an anomaly.
For example,
Chang's prototype generators~\cite{chang74} replaced training sets
of size \mbox{$T=(514,150,66)$} with prototypes of size \mbox{$N=(34,14,6)$}  (respectively).
That is,  prototypes may be as few as $\frac{N}{T}=(7,9,9)\%$ of the original
data. 
\subsection{Geometry of Instance Space}
What do these results say about the
distribution of project data in instance space?
We argue that,
fundamentally, $locality(1)$ is an assumption about the position of
examples within instance space. Consider the circles of \fig{dim}-
these are projects described in terms of a two dimensional instance
space (labeled here as $x$ and $y$).  Note that (a) the circles are
arranged (approximately) parallel to the $y$ axis and that (b) the
longer projects (indicated with larger circles) occur at higher $y$
values. The space of the circle examples in \fig{dim} could be
processed by $locality(1)$ since a single feature (in this case, the
vertical $y-axis$) usefully divides the examples with higher effort
from those with lower effort.
\begin{figure}[!h]
\begin{minipage}{1in}
\includegraphics[width=1in]{xy}
\end{minipage}~~~~
\begin{minipage}{1.6in}
{\em
Squares and circles show examples from two different projects.

The size of each space  shows the effort required for that project.
}
\end{minipage}
\caption{Projects described in two dimensions.}\label{fig:dim}
\end{figure}

Now consider the squares in \fig{dim}. As before, the size
of the shapes  indicates the effort associated with each project.
Note these examples do not parallel to either feature and the
longer and shorter projects are not easily separated by either axis.
The space of square circles require $locality(N)$: CART would need to
build multiple branches to capture the different regions of differing
effort; $k$-th nearest neighbor would need to use all features
and  $k=1$ in order to avoid learning from conflicting
examples (e.g. not two neighboring examples with very different efforts).


The results of this paper can be summarized as follows.
Given the idiosyncrasies of software development, we find it
unlikely that naturally occurring project examples will all line
up in a row parallel to one axis. Hence,
$locality(N)$ is required
to divide complex spaces into local regions of neighboring similar properties.

\section{Threats to Validity}
\label{sec:threats-to-validity}
{\em External validity} questions whether the results can be generalized outside
the specifications of a study\cite{Milic2004}.  
For the purpose of external validity, we use of 21 within-cross dataset pairs.
Among 10 studies investigated by Kitchenham et al. in~\cite{Kitchenham2007}, 9 of them used single within-cross dataset pairs, and 1 study used 6 pairs.
In terms of external validity, this report has higher validity than a standard \textit{within} vs. \textit{cross} data comparison effort estimation study.

Another consideration for external validity is the employed methods.
%Due to the nature of our research questions, particularly regarding selection tendency, we utilized ABE methods.
There are thousands of possible ABE variants and there is no way that this study covers them all.
There is obviously need for future research that repeats these experimentations with different ABE variants.
However, experiments reported here include a base variant (ABE0) as well as a filtering based variant (TEAK) run on 21 within-cross pairs.
Therefore, the extent of the experimentation in this research offers enough support for the claims that 1) \textit{cross} data performs no worse than \textit{within} data and 2) a \textit{within} test instance tends to select equally from \textit{within} and \textit{cross} projects. 


{\em Construct validity} (i.e. face validity) asks if we are measuring
what we actually intended to measure~\cite{Robson2002}.
Previous studies have concerned themselves with the construct validity of 
different performance measures for effort estimation (e.g. \cite{Stensrud2002}).
So as not to bias our conclusions due to a limited number of measures, we used $4$ different performance measures aided with win-tie-loss statistics.
%Another threat to construct validity is the formation of our \textit{cross-within divisions}.
%The proposed division criteria for \textit{within} and \textit{cross} data may as well define different corporations in real world.
%Also the use of available public data promotes the reproducibility of the conclusions of a research.
%However, we acknowledge that using \textit{within} and \textit{cross} data coming from completely different organizations would be a better option.
%On the other hand, the issue with using such proprietary data is the fact that they are difficult to be shared with research community and therefore proposed results cannot be reproduced, validated or refuted. 

In terms of {\em internal validity} of our results, there is one dimension of experimental conditions not explored.
We are making use of LOOCV, whose a possible alternative would be N-Way cross-validation.
In N-Way cross-validation, data is randomly divided into $B$ bins and each  bin is tested on a model learned from the combination of other bins (typical values for $B$ are 3 or 10).
From a theoretical point of view, not controlling the stability of our results across different testing strategies is a threat to validity, as different testing strategies entails different bias and variance conditions~\cite{Hastie2008}.
Elsewhere~\cite{Kocaguneli2011a}, we show that there is very little difference in the bias and variance values generated for LOOCV and N-way cross-validation. 
Since two testing strategies have similar bias-variance characteristics for effort datasets, we opted for LOOCV due to the fact that LOOCV is a deterministic procedure that can be exactly repeated by any other researcher with access to a particular data set.  
N-way cross-validation on the other hand requires a random number generator and a stratification heuristic (to maintain same class distribution in each bin).
Without access to exact same random number generator and stratification heuristic, it would be difficult for a researcher ``A'' to reproduce results of researcher ``B''.

\section{Conclusion}
We have shown that
when
using an effort estimator that is as good or better
than commonly used effort estimators (such as TEAK),
then if that estimator can select from $cross$ or $within$
soruces then:
\be
\item
The $cross$ performance results are no worse than $within$
(see \fig{cc-wc-performance-teak});
\item
The probability that the estimator selects a training instance
from $cross$ or $within$ is the same (see \fig{teak-perc}.b).
\ee


Result \#1 grants us permission to compare cross-vs-within results
(since there is no performance delta between them).
Result \#2 shows that the single-feature divisions have no
bearing on effort estimation.
Coupled with the results of~\cite{keung11},
these results are strong support for $locality(N)$ since we have
confirmed both {\bf PREDICTION~1} and {\bf PREDICTION~2}.

This means that (to  repeat a comment made in our introduction),
the most similar software to what you are writing now may not be in the next office. 
Rather, it may be in an office on the other side of the world.
As shown here,
using instance selection tools like TEAK, it is possible to automatically 
find that relevant training data.


%As to our research questions:
%
%\textbf{RQ1: What can be said about \textit{within} and \textit{cross}
%data performances?} The results of TEAK experiments confirm the
%findings of prior work~\cite{Turhan2009, Kocaguneli2010} on a larger
%scale: After relevancy filtering \textit{cross} data performance
%is comparable to that of \textit{within} data. 
% 
%\textbf{RQ2: What is the selection tendency of \textit{within} instances?}
%A surprising result of this research is the fact that under different
%ABE models (TEAK, ABE0, log+ABE0) test instances selected analogies
%from \textit{within} as well as \textit{cross} data.  This may be
%interpreted as a hint that organizations with very limited
%\textit{within} data can incorporate \textit{cross} data into their
%database.
%
%\textbf{RQ3: Which features are likely to define \textit{cross} and \textit{within} boundaries?}
%Our aim with this research question was to find features that would define boundaries to separate \textit{within} and \textit{cross} data from one another so that test instances would select mostly from its related \textit{within} dataset.
%However, none of the features we have identified resulted in such a scenario.
%On the contrary, they defined links between \textit{cross} datasets so that test instances selected from all subsets.

\section{Future Directions}
Some of the most likely future directions to this research are:
\bi
\item Reproduction of this work on proprietary data.
\item Investigating why particular subsets (cocomo81s, desharnaisL1) favor \textit{within} data, whereas the rest favors both \textit{within} and \textit{cross}.
\item Using different ABE or non-ABE methods under similar settings.
\item Experimenting if limited \textit{within} data can be supplemented with \textit{cross} data.
\item Using different features on different datasets to see if they can define a border between \textit{within} and \textit{cross} data.
\ei


\bibliographystyle{abbrv}
\bibliography{timm,library}

\balancecolumns
% That's all folks!
\end{document}


While it was not their intention to do so,
a recent ESEM paper on algorithmic methods
by Mair and Shepperd~\cite{Mair05}
comments on  {\bf PREDICTION~1}.
That paper searched the literature
for consistent results regarding the value
of regression or analogy-based methods for software effort estimation. 
If Mair and Shepperd had found that analogy always out-performs regressions, 
then that would have
been strong support for {\bf PREDICTION~1} since 
(a) regression builds one model across all variables;
and (b)
analogy-based methods  
use the $k$-thh nearest neighbors discussed in the introduction; i.e. they make estimates
using the subset of training data that clumps nearest the test instance.
However, their results were otherwise:
from a total of 20 empirical
studies, seven favored regression, four were indifferent and nine favored analogy.

Note that Mair and Shepperd did not re-run experiments: rather, they commented
on the consistency of empirical results generated by other researchers.
Keuing et al.~\cite{keung11} lament the state of empirical methods
in effort estimation. It is well established that any experiment ordering $M$ effort estimators
will generate different rankings, depending on the accuracy statistic used in the 
comparison; the random number generators used to build training and test sets;
and the data sets used~\cite{shepperd2003}. Unless these factors are controlled,
we must expect inconsistent results such as those reported
by Mair and Shepperd.

In an attempt to improve the state of empirical methods, Keung et al. explored the size
of the 
