\documentclass{acm_proc_article-sp}

\usepackage{times}
\usepackage{cite}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{multirow}
\usepackage{rotating}
\usepackage{subfigure}


%\usepackage[none]{hyphenat} 
\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}%
  \setlength{\partopsep}{0pt}%
  \setlength{\parskip}{0pt}%
  \begin{itemize}%
  \setlength{\leftmargin}{.2in}%
  \setlength{\parsep}{0pt}%
  \setlength{\parskip}{0pt}%
  \setlength{\itemsep}{0pt}}%
 {\end{itemize}}

\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
   \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}


\usepackage[table]{xcolor}
\usepackage{url,graphicx}
\newcommand{\G}{\cellcolor[rgb]{0.8,0.8,0.8}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\hyp}[1]{Hypothesis~\ref{hyp:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}

\newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
\newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}

\begin{document}

\title{How to Find Relevant Data for Effort Estimation?} 


\numberofauthors{2} %  in this sample file, there are a *total*
% of EIGHT authors. SIX appear on the 'first-page' (for formatting
% reasons) and the remaining two appear in the \additionalauthors section.
%
\author{
% 1st. author
\alignauthor
Ekrem Kocaguneli\\
%	   \titlenote{Dr.~Trovato insisted his name be first.}\\
       \affaddr{Lane Department of Computer Science and Electrical Engineering}\\
       \affaddr{West Virginia University}\\
       \affaddr{Morgantown, USA}\\
       \email{ekocagun@mix.wvu.edu}
% 2nd. author
\alignauthor
Tim Menzies\\
%\titlenote{The secretary disavows
%any knowledge of this author's actions.}\\
       \affaddr{Lane Department of Computer Science and Electrical Engineering}\\
       \affaddr{West Virginia University}\\
       \affaddr{Morgantown, USA}\\
       \email{tim@menzies.us}
}
%\date{30 July 1999}
% Just remember to make sure that the TOTAL number of authors
% is the number that will appear on the first page PLUS the
% number that will appear in the \additionalauthors section.

\maketitle
\begin{abstract}
~\\
\textit{\textbf{Background}}: Building effort estimators requires
the correct selection of training data.  How to find that data?
Generating a database of past projects \textit{within} an organization
requires considerable investment of 1) time, 2) money and 3) educated
personnel.  Hence, it is tempting to \textit{cross} the boundaries
of development type, location, language, application and hardware
to use existing datasets of other organizations.
\\
\textit{\textbf{Aim}:} Prior results are inconclusive about the
merits of cross-company data.  We conjecture that {\em instance
selection} can automatically find the instances most relevant, from
either within or acrcoss source dividisions.
\\
\textit{\textbf{Method}:} We filtered out 8 \textit{cross-within
divisions} (21 pairs of \textit{within-cross} subsets) out of 19
datasets and evaluated these divisions under different analogy-based
estimation (ABE) methods.
\\
\textit{\textbf{Results}:} The probablity of selecting training
data from within or across divisions based on single features is
the same.
\\
\textit{\textbf{Conclusion}:}
\end{abstract}


\category{H.4}{Software Cost Estimation}{\textit{k}-NN}
\category{D.2.8}{Software Engineering}{Cost}[within company, cross company]

\newtheorem{fore}{{\bf PREDICTION}}

\input{introduction}

\section{Related Work}
\subsection{Evidence for PREDICTION 1}

Keung et al.~\cite{keung11} built 90 effort estimators using
10 pre-processors and
9 learners.
Pre-processors included normalization, various descreitzaion methods,
feature selection (PCA, stepwise, sequential forward). 
Learners included  $k=1$ and $k=5$-nearest-neighor, linear and step-wise regression, 
CART, neural nets, \& PCR.

The 90 estimators were assessed via multiple accuracy statistics.
Let  $t$ instances have actuals $a_1,a_2,..,a_t$.
Prediction models generate prediction $p_i$ for  instance $i$.
If $|p_i - a_i|$ is $AR_i$ (the  {\em absolute residual}  difference between  predictions and actuals)
then:
\bi
\item
$MAR$ is the {\em mean absolute residual} $(\sum_i p_i - a_i)/t$.
\item
$MRE_i$ and $MER_I$ are the ratios  
$AR_i/a_i$ and 
$AR_I/p_i$.
\item
$PRED(25)$ is the percent of $t$ with $MRE_i \le 25\%$.
\item
$MMRE =\sum_i MRE_i/t$
\item
$MMER = \sum_i MER_i/t$.
\item
$MdMRE$ is median $MRE_i$ value. 
\item
Finally, the {\em balanced errors} are\newline
\mbox{$MBRE_i = (p_i - a_i )/ min(p_i,a_i)$} and\newline 
\mbox{$MIBRE_i = (p_i - a_i )/ max(p_i,a_i)$}
\ei
The 90 estimators were executed in a leave-one-our study
on  twenty data sets from http://promisedata.org/?cat=14
and their results compared using a Wilcoxon test (95\% confidence).
In a result that might have been predicted by Shepperd et
al.~\cite{Shepperd2002}, the ranking of many of the estimators varied
across the different data sets and the different accuracy estimators.

However, Keung et al. found a small group of 13 estimators that were
consisteny the best performers across all data sets (measured
according to all of MAR, MRE, PRED(25), MMRE, MMER, MdMRE, MBRE, and
MIBRE).  
In terms of this paper, 
the major result of Keung et at. is that all these 13 estimators
used CART or $k=1$ nearest neighbor. This is significant since
both these estimators use multiple features to sub-divide the training data:
\bi
\item
$k$-th nearest neighbor algorithms use all project features (perhaps, weighted by some
feature) to determine related projects~\cite{shepperd97};
\item
Tree-based algorithms like CART~\cite{breimann84} divide data into mulitple
braches  where each branch tests and divides that data on
multiple features.
\ei
Hence, this result is  strong support for {\bf PREDICTION~1}.

%Software effort estimation
%research falls into two broad groups:
%algorithmic and non-algorithmic methods; 
% a.k.a. expert-based estimation.
%Expert-based methods
%are often 
%human-intensive processes of estimate negotiation between domain experts~\cite{Jorgensen2004}.
%These negotiations continue until a concensus is reached among the experts.
%Possible pitfalls related with this family of methods include:
%\be
%\item obvious need for high-quality experts, 
%\item poor cabability of humans to improve their expert judgment skills~\cite{Jorgensen2009}  and 
%\item possible conflict of interest among domain experts~\cite{Kocaguneli2010}.
%\item difficulty with recreating and auditing prior estimates. This
%is a particular problem for, say, US Government projects that require all estimates be
%auditable by some external examiner at some future date~\cite{Menzies2006}.
%\ee
%These pitfalls with expert-based methods motivate us
%to explore algorithmic methods.
%Effort estimates generated by algorithmic methods are the product
%of a model that is built on historical data.  Such methods may
%entail the adaptation of an expert proposed model to historical
%data.  A very well known example to that scenario is Boehm's COCOMO
%method~\cite{Boehm1981}.  However, this option requires a long time
%interval for \textit{within} data collection and model adaptation
%to local data.  An alternative is the processing of historical data
%by generic methods.  Regression methods~\cite{Menzies2006}, neural
%nets~\cite{Kultur2008}, model trees~\cite{Stewart2002} and
%instance-based models~\cite{Menzies2006, Kocaguneli2010, Kadoda2000}
%are examples to this category.\footnote{what does this all mean?}
%
\subsection{Other Evidence}


Other results in the litereature are also inconclusive about  $locality(N)$.
In their review of papers building effort models using data from within one
company or across
mulitple companies, Kitchenham et al.~\cite{Kitchenham2007} found  equal evidence for and against the value of
building effort models based on a single feature division (specifically, they
found four studies favoring the use of \textit{within} compnay
data, and  three reports that  using
\textit{cross} data performance is not significantly worse than \textit{within}). 
In other work, in the field of defect prediction,
Zimmermann et al.~\cite{Zimmermann2009} found that predictors
performed worse when trained from cross-application data than from within-application data. The evidence for their conclusion is quite emphatic:
{\em within} defeated {\em cross} in
618 out of 622 comparisons.

On the other hand, support for $locality(N)$
comes from the work of 
of Turhan et al.~\cite{Turhan2009},
and 
Kocaguneli et al.~\cite{Kocagunel2010}.
Turhan et al. compare defect predictors learned from cross or within company data.
Like Zimmermann, they found that using {\em all} cross company data lead to poor predictor
preformance (very large false alarm rates). However, after {\em instance selection} pruned
away irrelevant cross company data, they found that the cross company predictors were
equivalent to the predictors learned from within company data~\cite{Turhan2009}.
Inspired by Turhan et al.,
Kocaguneli et al.~\cite{Kocagunel2010} used {\em instance selection}
as a pre-processor for a study on cross-vs-within company effort estimation.
In a limited study with
three data sets, they found that after pruning away irrelevant instances, 
the performance differences in
the predictors learned from cross or within data were statistically insignificant.

\subsection{Resolving the Evidence}

The Turhan, Kocaguneli et al. results clearly support $locality(N)$ but the other
results discussed above are either inconsistent or unsupportive.
How are we to reconcile this conflicting evidence?
One way is to note that:
\bi
\item Studies supporting $locality(N)$ all used instance selection.
\item With one exception, instance selection is {\em not} seen in the
Kitchenham, Zimmermann et al. studies.\footnote{ekrem: check}
\ei
The one exception is that 
one study survey by  Kitchenham et al. explored  analogy-based estimation
and, in that analogy-based result, training on cross company data worked as well as 
training on within company data. Analogy-based methods use every feature (perhaps, with
some feature weighting) to find relevant training examples.
Hebnce, this exception  offers more  support
for  $locality(N)$ than for $locality(1)$.

This single example from Kitchenham et al., plus the studies with three 
data sets found in~\cite{Kocagnueli2010}
are hardly conclusive since they do not collect the information required to
comment on {\bf PREDICTION~2}. What is required is a well-controlled
 instance selection
experiment over 
data divided by using some single feature. {\bf PREDICTION~2} would be supported
if the instance selector selects as much data {\em within} as {\em across}
the single feature divisions. 

\fig{problem-types} illustrates how this study differs to prior work.
Most effort estimation research falls into \fig{problem-types}.a where 
estimation models are applied {\em within} one source set to learn an estimator.
Examples of this approach include \cite{Mendes2003,Auer2006,Li2009,Shepperd97,Menzies2006}.

A smaller number of papers, such as those surveyed by Kitchenham et al.~\cite{Kitchemham2007},
explore building models using data that falls {\em across} many sources (see \fig{problem-types}.b).
Fewer still are the papers like ~\cite{Turhan2009,Kocagnueli2010}).
that, prior to learning,
apply some instance selection filter to cross company data
souces (see \fig{problem-types}.c).

To the best of our knowledge, this paper is the first that allows an effort estimator 
to select training data from  either {\em cross} or {\em within} different sources,
then checks what data was selected from
which source  (see \fig{problem-types}.d).

\begin{figure}[t]
  \centering                
  \subfigure[\textit{Within}]{\label{fig:within-problem}\includegraphics[width=0.09\textwidth]{withinProblem.png}}
  \subfigure[\textit{Cross}]{\label{fig:cross-problem}\includegraphics[width=0.09\textwidth]{crossProblem.png}} 
  \subfigure[\textit{Filter}]{\label{fig:cross-filter-problem}\includegraphics[width=0.09\textwidth]{crossFilterProblem.png}} 
  \subfigure[\textit{Within+Cross}]{\label{fig:within-cross-problem}\includegraphics[width=0.18\textwidth]{withinCrossProblem.png}} 
  \caption{The problem types in \textit{within} vs. \textit{cross} data comparison and our conclusions so far. \textit{``+''} and \textit{``-''} signs on top of each approach
 mean positive and negative results respectively. A \textit{``?''} sign means that
the approach  has not previously been investigated.}
  \label{fig:problem-types}
\end{figure}



\subsection{Research Questions}
To guide this research the following research questions are defined:
\bi
\item[RQ1:] What can be said about \textit{within} and \textit{cross} data performances?
\item[RQ2:] What is the selection tendency of \textit{within} instances?
\item[RQ3:] Which features are likely to define \textit{cross} and \textit{within} boundaries?
\ei




%\subsection{Contributions}
%The contributions of this research can be listed as follows:
%\bi
%\item Analysis of effort datasets and proposition of subsets for possible WC-CC experiments.
%\item Investigating effort dataset features  to see which features are likely to define \textit{cross} and \textit{within} boundaries.
%\item An extensive analysis on the merits of \textit{cross} data and uniform conclusions.
%\item Proposing a mixture model of \textit{within} and \textit{cross} data.
%\item Investigating tendency of test instances towards \textit{cross} and \textit{within} data to find preferable percentage of a mixture.
%\ei
%
%

\section{Methodology}

The details of dataset filtering and division of them into \textit{cross-within divisions} are presented here.
We also explain the selected performance measures and our experimental set-up.

\subsection{Datasets}
There are 2 fundamental factors that were considered for selection of the datasets used in this research:
\bi
\item Public availability: For reproducibility purposes
\item Cross-within divisibility: For enabling \textit{cross} vs. \textit{within} experimentation
\ei
A critical issue in software engineering is the ability of the proposed results to be reproducible~\cite{Juristo2009, Kitchenham2007} and use of proprietary data is a major obstacle towards this goal.
Therefore, all our datasets are publicly available through PROMISE data repository~\cite{Boetticher2007}.

We define \textit{cross-within division} as the subset(s) of effort
data that are formed through division of one feature:
instances having the same value for that feature  form a
subset.  This feature are plausible candidates for generating 
a \textit{cross} source experiment, i.e. the features should be
likely to change from one source to other.
Accordingly,
this study began by exploring what PROMISE effort data
can be divided via a single feature.
After manually inspecting more than 20 datasets,
six were found to be suitable for \textit{cross-within} experimentation.
Those six data sets support the
21
\textit{cross-within divisions} shown in \fig{cc-selection}.
The selected division criteria include:
\bi
\item project type: embedded, organic and semidetached (\textit{cocomo81}), 
\item center: geographical development center (\textit{nasa93}), 
\item language type: programming language used for development (\textit{desharnais}), 
\item application type: on-line service program, production control program etc. (\textit{finnish} and \textit{maxwell}), 
\item hardware: PC, mainframe, networked etc. (\textit{kemerer} and maxwell), 
\item source: whether in-house or outsourced (maxwell).
\ei
In the following, we will use the following nomecluteres.
If we name a data set, followed by a set of numbers,
these
correspond to values of the  feature used to form the subsets.
If a name has multiple numbers at the end (e.g. finnishAppType2345) this means that all instances with these  feature values are combined in a single subset.
Also note that the terms \textit{within/cross} and WC/CC will be used interchangeably for the rest of the text.

\begin{figure}
\scriptsize
\center
\begin{tabular}{l | l | l | r}
\textbf{Dataset} & \textbf{Criterion} & \textbf{Subsets} & \textbf{Subsets Size}\\[1ex]\hline\hline
cocomo81 & project type & cocomo81e &	28  \\
  & & cocomo81o   & 	24  \\
  & & cocomo81s   & 	11	  \\\hline
  nasa93 & development center & nasa93\_center\_1   & 	12  \\
  & & nasa93\_center\_2   & 	37  \\
  & & nasa93\_center\_5   & 	39  \\\hline	
  desharnais & language type & desharnaisL1   & 	46  \\
  & & desharnaisL2   & 	25  \\
  & & desharnaisL3   & 	10  \\\hline
finnish & application type & finnishAppType1 & 17 \\
 &  & finnishAppType2345 & 18 \\\hline
kemerer & hardware & kemererHardware1 & 7 \\
 &  & kemererHardware23456 & 8 \\\hline
maxwell & application type & maxwellAppType1 & 10 \\
 &  & maxwellAppType2 & 29 \\
 &  & maxwellAppType3 & 18 \\\hline
maxwell & hardware & maxwellHardware2 & 37 \\
 &  & maxwellHardware3 & 16 \\
 &  & maxwellHardware5 & 7 \\\hline
maxwell & source & maxwellSource1 & 8 \\
 &  & maxwellSource2 & 54 \\
\end{tabular}\caption{
6 datasets are selected from 20+ candidates.
Then selected datasets are divided into subsets according to a criterion that can define a \textit{cross-within division}. The datasets, subset sizes as well as the selection criteria are provided here.}
\label{fig:cc-selection}
\end{figure}

%We acknowledge that \textit{cross-within division} is a validity concern (a detailed discussion is given in \tion{threats-to-validity}).
%However, we hasten to say that it is an acceptable experimentation method, because: a) Despite all our efforts, public effort datasets are still limited; b) For a comprehensive analysis (as this study does on 21 subsets), alternative use of available datasets should be considered.

\subsection{Instance Selection}
The goal of our experiment is to determine at what probability
an  agent has selects
training instances from
either cross- or within- sources. The instance selector used in this
study is 
TEAK~\cite{Kocaguneli2011}, which is best explain as an 
exxtension to ABE0~\cite{Kocaguneli2010,Kocaguneli2011}.

\subsubsection{ABE0}

Analogy-based estimators
(ABE) generate an estimate for a \textit{test} project by retrieving similar past projects (a.k.a. analogies) from a database of past projects and adapting their effort values into an estimate.
We use ABE is this study since 
they are  a 
widely investigated method in the literature~\cite{Mendes2003, Chang1974, Keung2008a, Li2009, Kadoda2000,Kocaguneli2010,Kocaguneli2011}, 2) they are particularly helpful for \textit{cross} source
studies as they are based on distances between individual project instances.

There are various design options associated with ABE methods such as the distance measure for nearness~\cite{Mendes2003}, adaptation of analogy effort values~\cite{Mendes2003}, row processing~\cite{Chang1974, Keung2008a}, column processing~\cite{Keung2008a, Li2009} and so on.
Elsewhere~\cite{Keung2011} we show that these options can easily lead to more than 6000 ABE variants. 

ABE0 is a {\em baseline} analogy-based estimator which combines
the ABE tools used in Kadoda \&
Shepperd~\cite{Kadoda2000},  Mendes et al.~\cite{Mendes2003}, and Li et al.~\cite{Li2009}:
\bi
\item Input a database of past projects, whose rows are projects instances and whose columns are \textit{independent} variables (that define projects) and a \textit{dependent} variable (effort value). 
\item Decide how many similar projects (\textit{analogies}) are to be used from the training set, i.e \textit{k}-value.
\item For each test instance, retrieve \textit{k} analogies from the database.
\bi
\item For selection of \textit{k} analogies use a similarity measure like Euclidean distance measure.
\item Before calculating similarity, scale independent features to equalize their influence on the similarity measure.
\item Use a feature weighting scheme to reduce the effect of less informative features.
\ei
\item Adapt the effort values of the \textit{k} nearest analogies to come up with the effort estimate.
\ei
ABE0 uses the Euclidean distance as a similarity measure, whose formula is given in Equation \ref{equ:euclid}, where $w_i$ corresponds to feature weights applied on independent features.
ABE0 framework does not favor any features over the others, therefore each feature has equal importance in ABE0, i.e. $w_i=1$.
For adaptation ABE0 takes the median of selected \textit{k} projects.

\begin{equation}\label{equ:euclid}
Distance = \sqrt{\sum_{i=1}^{n}{w_i(x_{i} - y_{i})^{2}}}
\end{equation}

\subsubsection{TEAK}

TEAK is an {\em variance-based} instance selector that discards training data
associated with regions of high estimation variance.
It augments ABE0 with instance selection and an indexing scheme for finding
relevant training examples.
Detailed description of TEAK can be found in~\cite{Kocaguneli2011}.

In summary, TEAK is a two-pass system:
\bi
\item Pass 1 prunes training instances implicated in poor decisions;
\item Pass 2 selects instances closest to the test instance.
\ei
In the first pass, training instances are combined using greedy-agglomerative clustering,
or GAC, to form an initial cluster tree we call  GAC1; e.g.
\fig{gac-tree}.
Level zero of GAC1 is formed by leaves, which are the individual project instances.
These instances are greedily combined into tuples to form the nodes of upper levels.
The variance of the effort values associated with each sub-tree (the performance variance)
is then recorded and normalized $min .. max$ to $0 .. 1$.
The high variance sub-trees are then pruned, as these are the sub-trees that would 
cause an ABE method to make an estimate from a highly variable instance space. 
Hence, pass one prunes sub-trees with a variance greater than $\alpha\%$
of the maximum variance seen in any tree.  After some experimentation,
we found that $\alpha=10$ lead to estimates with lowest errors.

The leaves of the remaining sub-trees are the {\em survivors} of pass one.
They are filtered to pass 2 where they are used to build a second GAC tree (GAC2).
GAC2 is generated and traversed in a similar fashion to GAC1, then test instances are moved from root to leaves.
Unlike GAC1, this time variance is a decision criterion for the movement of test instances: If the variance of the current tree is larger than its sub-trees, then continue to move down; otherwise, stop and select the instances in the current tree as the analogies.  
TEAK is a form of ABE0, so its adaptation method is the same, i.e. take the median of the analogy effort values.
A simple
visualization of this approach is given in in
\fig{teak}.


\begin{figure}[!t]
\begin{center}
\includegraphics[width=0.35\textwidth]{gac-tree.png}
\end{center}
\caption{A sample GAC tree with regions of high variance (red) and low variance (green). GAC trees may not always be binary. For example here, leaves are odd numbered, hence node ``c'' is left \textit{behind}. Such instances are pushed \textit{forward} into the closest node in the higher level. For example, ``c'' is pushed forward into the ``b+f'' node to make ``b+f+c'' node.  
}\label{fig:gac-tree}
\end{figure}


\begin{figure}[!t]
\begin{center}
\includegraphics[width=0.35\textwidth]{teak.png}
\end{center}
\caption{Execution of TEAK on 2 GAC trees, where tree on the left is GAC1 and the one on the left is GAC2 (i.e. lower variance sub-tree of GAC1). The instances in the low variance region of GAC1 (green region) are selected to form GAC2. Then test instance traverses GAC2 until no decrease in effort variance is possible. Wherever the test instance stops is selected as the subtree to be used for adaptation (white region of GAC2).   
}\label{fig:teak}
\end{figure}

\begin{figure}[!th]
\renewcommand{\baselinestretch}{.5}
\centering
\small
  \begin{tabular}{lccccccccc}
& & & &\multicolumn{5}{c}{~~~~~~~~~~Simple ABE0}\\\cline{6-10}
&	\begin{sideways}\parbox{9mm}{TEAK}\end{sideways}	&	\begin{sideways}\parbox{9mm}{LR}\end{sideways}	&	\begin{sideways}\parbox{9mm}{NNet}\end{sideways}	&	\begin{sideways}\parbox{9mm}{\textit{k=best}}\end{sideways}	&	\begin{sideways}\parbox{9mm}{k=1}\end{sideways}	&	\begin{sideways}\parbox{9mm}{k=16}\end{sideways}	&	\begin{sideways}\parbox{9mm}{k=2}\end{sideways}	&	\begin{sideways}\parbox{9mm}{k=4}\end{sideways}	&	\begin{sideways}\parbox{9mm}{k=8}\end{sideways}	\\\hline
\multicolumn{10}{l}{  \underline{{\bf MdMRE}}}	\\
Cocomo81	   &	$\blacktriangle$ 	&		&		&	&		&		&		&		&		\\
Cocomo81e	    &	$\blacktriangle$ 	&		&		&		&		&		&		&		&		\\
Cocomo81o	    &	$\blacktriangle$ 	&		&		&		&		&		&		&		&		\\
Nasa93	        &	    &$\blacktriangle$ 	&	&		&		&		&		&		&		\\
Nasa93c2	    &	     &$\blacktriangle$	&	&		&		&		&		&		&		\\
Nasa93c5     	&	$\blacktriangle$ 	 &		&	&		&		&		&		&		&		\\
Desharnais	    &		     &$\blacktriangle$	&		&		&		&		&		&		&		\\
Sdr	            &	$\blacktriangle$	  &		&		&		& 	&		&		&		&		\\
ISBSG-Banking	&	$\blacktriangle$ 	 &		&		&		&		&		&		&		&		\\
\em Count	&	6	&	3	 &	0	&	0	&	0	&	0	&	0	&	0	&	0	\\


\hline\multicolumn{10}{l}{  \underline{{\bf Pred(25)}}}	\\
Cocomo81	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Cocomo81e	&		&		&	$\blacktriangle$	&		&		&		&		&		&		\\
Cocomo81o	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Nasa93	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Nasa93c2	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Nasa93c5	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Desharnais	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Sdr	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
ISBSG-Banking	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
\em Count	&	5	&	3	&	1	&	0	&	0	&	0	&	0	&	0	&	0	\\


\hline\multicolumn{10}{l}{ \underline{{\bf MAR}}}	\\
Cocomo81	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Cocomo81e	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Cocomo81o	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Nasa93	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Nasa93c2	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Nasa93c5	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Desharnais	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Sdr	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
ISBSG-Banking	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
\em Count	&	6	&	3	&	0	&	0	&	0	&	0	&	0	&	0	&	0	\\
\hline
\end{tabular}
\caption{
Restuls fro
20 repeats
of a leave-one-out experiment,
repeated for the performance measures of MdMRE, Pred(25) and MAR.
Black triangles mark when an estimator was one of the top-ranked methods
for a particular data set (where ranking was computed via $win-loss$ from a WILCOXON test, 95\% confidence). 
The $Count$ rows show the number of times a method appeared as the top performing variant.
Results from~\cite{Kocaguneli2011}.
}
\label{fig:abe-vs-non-abe}
\end{figure}


We use TEAK in this study since, as shown by leave-one-out
experiments
of 
~\cite{Kocaguneli2011},
its performance is comporable 
with other commonly-used effort estimators
including neural networks (NNet) and linear regression (LR).
A summary of those performance results are 
given in \fig{abe-vs-non-abe} (for a complete analysis and for definitions of datasets please refer to Figure 7 of~\cite{Kocaguneli2011}).
That figure shows the results of a statistical comparison of various performance indicators (defined in \fig{keung1})
for nine effort estimators and nine data sets from http://promisedata.org/?cat=14:
\bi
\item The columns $k=1,2,4,8,16$ denote variants of standard ABE0  where estimates are generated from
the $k$-th nearest neighbors.
\item The column $k=best$ denote a variant of ABE0 where $k$ was choosen by an initial pre-processor
that selected a best $k$ value after exploring the training data.
\item The columns $LR$ and $NNet$ refer to linear regression and neural nets.
\ei
The black triangles in \fig{abe-vs-non-abe} mark when an estimator was one of the top-ranked methods
for a particular data set. Ranking was accomplished via the $win-loss$ calulation of \fig{win-loss-tie-loss-psuedo}.
We first check if  two distributions $i,j$ are statistically different according to the WILCOXON
test.
In our experimental setting, $i,j$ are arrays of performance measure results coming from two different methods.
If they are not statistically different, then they are said to \textit{tie} and we increment $tie_{i}$ and $tie_{j}$.  
On the contrary, if they are different, we updated $win_{i},win_{j}$ 
and ${loss_{i},loss_j}$ after a numerical comparison of performance measures.
The related pseudocode is given in \fig{win-tie-loss-pseudo}.
To get rid of any bias that would come from a particular experimental setting, for every experiment $20$ runs are made.

The key feature of \fig{abe-vs-non-abe} is that
TEAC always performed better than the other ABE0 methods, and usually  performed better than
 neural nets. TEAC's only
near-rival was linear regresison but, as shown in the $LR$ columns, TEAC was ranked top
nearly twice as much as linear regression.
 

%\subsection{Performance Measures}
%\label{sec:performance-measures}
%A performance measure comments on the success of an estimate, hence the predictor.
%Performance measures listed here have the property that there is at least one publication in effort estimation research proposing their use.
%An example performance measure is the absolute residual (AR), which is the absolute difference between predicted and the actual effort values.
%AR formula is given in \eq{ar}, where $x_i,\hat{x_i}$ are the actual and predicted values respectively for test instance $i$.
%Summary of individual AR values is found by taking their mean (MAR).
%
%{\begin{equation}\label{eq:ar}
%AR_i=|x_i-\hat{x_i}|\
%\end{equation}}
%
%Another performance measures is the Magnitude of Relative Error (MRE), which is a widely used method to select the best estimator from a number of competing models~\cite{Shepperd1997, Foss2003}. 
%MRE is the measure of the error ratio between the actual and the predicted effort:  
%{\begin{equation}\small\label{one}
%MRE_i=\frac{\mid{x_i-\hat{x_i}}\mid}{{x_i}} = \frac{\mid{AR_i}\mid}{{x_i}}
%\end{equation}}
%
%
%MRE can be summarized through mean or median.
%The former summary defines mean MRE (MMRE) and the latter defines median MRE (MdMRE).
%Formulas of MMRE and MdMRE are: 
%
%
%{\begin{eqnarray}\small\label{one}
%MMRE = mean(MRE_1, MRE_2, ... , MRE_n ) \\
%MdMRE = median(MRE_1, MRE_2, ... , MRE_n )
%\end{eqnarray}}
%
%An alternative to prior performance measures is $Pred(x)$, i.e. the percentage of estimates that fall within $x\%$ of the actual values:
%{\begin{equation}\small
%Pred(x) = \frac{100}{N}\sum_{i=1}^N\left\{\begin{array}{ll}
%\mbox{1 if $MRE_i \le \frac{x}{100}$}\\
%\mbox{0 otherwise}
%\end{array}\right.
%\end{equation} 
%}
%A common value for $x$ in $Pred(x)$ is $30$~\cite{Chulani1999}.
%For example, $Pred(30)=50\%$ implies that half of the estimates are within $30\%$ of the actual values \cite{Shepperd1997}.
%
%
\begin{figure}
\renewcommand{\baselinestretch}{1}
\begin{algorithmic} \footnotesize
\STATE \textit{$win_{i}$} = 0, \textit{$tie_{i}$} = 0, \textit{$loss_{i}$} = 0
\STATE \textit{$win_{j}$} = 0, \textit{$tie_{j}$} = 0, \textit{$loss_{j}$} = 0
\IF{WILCOXON(\textit{$P_i$}, \textit{$P_j$}) says they are the same}
	\STATE \textit{$tie_{i}$} = \textit{$tie_{i}$} + 1;
	\STATE \textit{$tie_{j}$} = \textit{$tie_{j}$} + 1;
\ELSE
	\IF{mean or median(\textit{$P_{i}$}) $<$ median(\textit{$P_{j}$})}
		\STATE \textit{$win_{i}$} = \textit{$win_{i}$} + 1
		\STATE \textit{$loss_{j}$} = \textit{$loss_{j}$} + 1
	\ELSE
		\STATE \textit{$win_{j}$} = \textit{$win_{j}$} + 1
		\STATE \textit{$loss_{i}$} = \textit{$loss_{i}$} + 1
	\ENDIF
\ENDIF
\end{algorithmic}
\caption{Pseudocode for win-tie-loss calculation between methods \textit{i} and \textit{j}
w.r.t. performance measures $P_i$ and $P_j$. Note here that only for Pred(30) the comparison is based on actual values ($Pred(30)_i$, $Pred(30)_j$) rather than mean or median values of performance measure arrays (median(\textit{$P_{i}$}), median(\textit{$P_{j}$})).}
\label{fig:win-tie-loss-pseudo}
\end{figure}
%It is reported as a considerable threat to the validity of effort estimation studies, if derived performance measures are not evaluated with an appropriate statistical test~\cite{Kitchenham2009}.
%In this study we use a Mann Whitney test (95\%).
%To compare the values of MAR, MMRE, MdMRE and Pred(30), we use the following win-tie-loss procedure that incorporates Mann Whitney statistical test.

\subsection{Experimentation}
The experimentation of this research has two different goals:
\bi
\item
The {\em   performance comparison}
goal compares different ABE methods (TEAK and ABE0 with
\textit{k={4,best}}) when subject to \textit{within} and \textit{cross}
sourcey data.
\item
 The {\em selection tendency}  goals question the
tendency of a \textit{within} test instance to selection \textit{within}
or \textit{cross} data.  In other words, given the chance that a
test instance had access to \textit{within} and \textit{cross} data
at the same time, what percentage of every subset would be selected
into \textit{k} analogies used for estimation?
\ei
\subsubsection{Performance Comparison}
For performance comparison we have two settings: \textit{Within}
and \textit{cross}.  In \textit{within} data setting, only
\textit{within} one source is used as the dataset and a testing strategy
of leave-one-out cross-validation (LOOCV) is employed.  LOOCV works
as follows: Given a \textit{within} dataset of $T$ projects, $1$
project at a time is selected as the test  and the remaining $T-1$
projects are used for  training, so eventually we have $T$ predictions.
The resulting $T$ predictions are then used to compute $4$ different
performance measures defined in \S2: PRED(30), MAR, MMRE, MdMRE. 

\textit{Cross} data setting uses \textit{within} data as the test
set and the \textit{cross} data as the training set.  In this setting
LOOCV is used as follows: each \textit{within} source is selected
as the test instance and TEAC methods derive an estimate for that
instance by adapting \textit{cross} analogies.  Ultimately we end
up with $T$ predictions adapted from a \textit{cross} dataset.
Finally the performances of TEAK and ABE0 methods under \textit{within}
and \textit{cross} data settings are compared.  For that purpose
we use both mere performance values as well as win-tie-loss statistics.

\subsubsection{Selection Tendency}
For selection tendency, 
we select test instances according to LOOCV.
For each test instance, we are left with training sets of $T-1$ \textit{within} data and the subsets of \textit{cross} data.
After marking every \textit{within} and \textit{cross} instance, we combine the two datasets into a single training set and let the test instance choose analogies from the unified training set.
In this setting our aim is to see what percentage of \textit{within} and \textit{cross} subsets would appear among selected \textit{k} analogies.
The percentage for a subset $S_i$ is calculated in accordance with \eq{percentage}\footnote{ekrem- simpler?
what is $S_i$}:

\begin{equation}\label{eq:percentage}
Percentage = \frac{\# of\; Instances From S_i In Analogies}{Size(S_i)}
\end{equation}

\section{Results}
\label{sec:results}
\begin{figure*}[th!]
\renewcommand{\baselinestretch}{0.8} 
\scriptsize
\centering
\begin{tabular}{l | l l l | l l | l l l | l l | l l l | l l | l l l | l l }
\textbf{Dataset} & \multicolumn{5}{c |}{\textbf{MAR}} & \multicolumn{5}{c |}{\textbf{MMRE}} & \multicolumn{5}{c |}{\textbf{MdMRE}} & \multicolumn{5}{c}{\textbf{Pred(30)}}  \\\hline
        & \textbf{W} & \textbf{T} & \textbf{L}  & \textbf{WC}  & \textbf{CC} & \textbf{W} & \textbf{T} & \textbf{L}  & \textbf{WC}  & \textbf{CC} & \textbf{W} & \textbf{T} & \textbf{L}  & \textbf{WC}  & \textbf{CC} & \textbf{W} & \textbf{T} & \textbf{L}  & \textbf{WC}  & \textbf{CC} \\
\input{winTieLossFileAll}
\end{tabular}
\caption{Results of TEAK: Comparison of performance between \textit{within} and \textit{cross} data w.r.t. 4 different performance measures (MAR, MMRE, MdMRE, Pred(30)) as well as \textbf{\textit{W, T, L}} statistics. Highlighted rows are the cases, where \textit{within} data is \textit{``dominantly''} better than \textit{cross}, i.e. wins more than half the time. Under the columns of \textbf{\textit{WC}} and \textbf{\textit{CC}} the actual performance values associated with \textit{within} and \textit{cross} source
 datasets are provided respectively.}
\label{fig:cc-wc-performance-teak}
\end{figure*}


\subsection{Performance Comparison}


For performance comparison 4 different performance measures are
employed: MAR, MMRE, MdMRE and Pred(30).  The actual performance
values are also evaluated subject to Mann Whitney\footnote{ekrem:
mann whitney? wilcoxon? why do we keep swapping between them?}
statistical test at $95\%$ confidence and this evaluation is
summarized by win-tie-loss statistics.  

\fig{cc-wc-performance-teak} shows \textit{within} and \textit{cross}
data performance when TEAK is used as the estimation method.  For
each performance measure win-tie-loss statistics  (abbreviated with
\textbf{W}, \textbf{T}, \textbf{L} respectively) of \textbf{WC}
performance when compared to \textbf{CC} over 20 runs as well as
actual performance measure values are reported.  

The gray lines in 
\fig{cc-wc-performance-teak} shows the experiments
where the $within$ results ``dominat'';
i.e.
win in more than half the comparisons.
Note that there are only  two gray lines.
In the remaining $\frac{19}{21}$ cases, the \textit{within} data
does not provide an advantage over \textit{cross} data.  In fact,
in one
case (kemererHardware1) the \textit{within} data is far
worse than \textit{cross} with an \textbf{L} value of 20.  These
results are confirmation of previous conclusions~\cite{Turhan2009,
Kocaguneli2010} in a much larger scale with $4$ error measures and
$21$ different cases: instance selection on \textit{cross} sources
improves its performance to an extent where it is no worse than
\textit{within} data.  


%Another intriguing question is what happens in this large-scale
%comparison, when we remove the relevancy filtering.  The performance
%comparison of \textit{within} and \textit{cross} data subject to
%estimates of ABE0 with \textit{k=\{4, best\}} is given in
%\fig{cc-wc-performance-knn}.  The cells of \fig{cc-wc-performance-knn}
%can have 3 values: \textit{``+'', ``-'' and ``o''.}. The \textit{``+''}
%sign tells that \textit{within} data performance is \textit{``dominantly''}
%better than that of \textit{cross} data, i.e. \textit{within} won
%more than 10 out of 20 runs, whereas a \textit{``-''} sign tells
%that \textit{within} lost more than 10 runs. If none of these
%conditions occur, i.e. \textit{within} and \textit{cross} performances
%tie, then a ``o'' sign is assigned to the cell.  The cases where
%\textit{within} data is \textit{``dominantly''} better than
%\textit{cross} are highlighted.
%
%\begin{figure}[th!]\renewcommand{\baselinestretch}{0.8}
%\scriptsize
%\begin{tabular}{l | l l | l l | l l | l l }
%\textbf{Dataset} & \multicolumn{2}{c |}{\textbf{MAR}} & \multicolumn{2}{c |}{\textbf{MMRE}} & \multicolumn{2}{c |}{\textbf{MdMRE}} & \multicolumn{2}{c}{\textbf{Pred(30)}}  \\\hline\hline
%         & \begin{sideways}\textbf{\textit{k}=4}\end{sideways} & \begin{sideways}\textbf{\textit{k}=best}\end{sideways} 
%         &  \begin{sideways}\textbf{\textit{k}=4}\end{sideways} & \begin{sideways}\textbf{\textit{k}=best}\end{sideways}  
%         & \begin{sideways}\textbf{\textit{k}=4}\end{sideways} & \begin{sideways}\textbf{\textit{k}=best} \end{sideways}
%         & \begin{sideways}\textbf{\textit{k}=4}\end{sideways} & \begin{sideways}\textbf{\textit{k}=best}\end{sideways} \\\hline
%\input{winTieLossKNN}
%\end{tabular}
%\caption{ABE0 performance comparison between \textit{within} and \textit{cross} data w.r.t. 4 different performance measures (MAR, MMRE, MdMRE, Pred(30)) with different \textit{k} values. Each cell in this table can have three values: \textit{``+'', ``-'' and ``o''.}. A \textit{``+''} sign indicates that \textit{within} performance is \textit{``dominantly''} better than \textit{cross}, i.e. it won more 10 of the 20 runs, whereas a \textit{``-''} sign tells that \textit{cross} lost more than 10 runs. If none of these conditions occur, i.e. \textit{within} and \textit{cross} performances tie, then a ``o'' sign is assigned to the cell. For convenience \textit{``+''} signs are highlighted.}
%\label{fig:cc-wc-performance-knn}
%\end{figure}

%Notice in \fig{cc-wc-performance-knn} that out of \textit{21 Subsets} $\times$ \textit{4 error measures} $\times$ \textit{2 ABE0 methods} = \textit{168 cases}, there are only $3$ cases where \textit{within} performance is dominantly better. 
%For the majority of the remaining cases, \textit{within} and \textit{cross} performance is comparable.
%Surprisingly for a minority of the remaining cases (see ``-'' signs), \textit{cross} data performance is dominantly better than that of \textit{within}.
%These results are important in the sense that  unlike previous work that report inconclusive results on the merits of cross data~\cite{Mendes2007, Zimmermann2009}; we are able to see enough uniformity in a large scale experiment with $168$ cases.
%The uniformity concludes that \textit{cross} data is as high performing as \textit{within} data.
%However, we should remember that such uniformity may come from \textit{cross-within divisions}, which assume \textit{cross} subsets of a division are all collected through the same methodology.
%%
%\begin{figure}[th!]\renewcommand{\baselinestretch}{0.8}
%\scriptsize
%\begin{tabular}{l | l l | l l | l l | l l }
%\textbf{Dataset} & \multicolumn{2}{c |}{\textbf{MAR}} & \multicolumn{2}{c |}{\textbf{MMRE}} & \multicolumn{2}{c |}{\textbf{MdMRE}} & \multicolumn{2}{c}{\textbf{Pred(30)}}  \\\hline\hline
%         & \begin{sideways}\textbf{\textit{k}=4}\end{sideways} & \begin{sideways}\textbf{\textit{k}=best}\end{sideways} 
%         &  \begin{sideways}\textbf{\textit{k}=4}\end{sideways} & \begin{sideways}\textbf{\textit{k}=best}\end{sideways}  
%         & \begin{sideways}\textbf{\textit{k}=4}\end{sideways} & \begin{sideways}\textbf{\textit{k}=best} \end{sideways}
%         & \begin{sideways}\textbf{\textit{k}=4}\end{sideways} & \begin{sideways}\textbf{\textit{k}=best}\end{sideways} \\\hline
%\input{winTieLossKNNLog}
%\end{tabular}
%\caption{\textit{log}+ABE0 performance comparison between \textit{within} and \textit{cross} data w.r.t. 4 different performance measures (MAR, MMRE, MdMRE, Pred(30)) with different \textit{k} values. The notation used here is the same as \fig{cc-wc-performance-knn}. The highlighted cells are the cases, where \textit{within} data is dominantly better than \textit{cross} data.}
%\label{fig:cc-wc-performance-knn-log}
%\end{figure}

%The pre-processors applied on the data before an estimation model can have a significant effect on the performance~\cite{Menzies2006}.
%In~\cite{Keung2011} we investigate the stability of rankings among $90$ different methods that include both analogy and non-analogy methods.
%We have seen that applying a \textit{log} transformation on the data as a pre-processing step to ABE0 (log+ABE0), can improve the ranking of ABE0 by orders of magnitude.
%The results of log+ABE0 is summarized in \fig{cc-wc-performance-knn-log}.
%The notation of \fig{cc-wc-performance-knn-log} is the same as that of \fig{cc-wc-performance-knn}.
%See that the general picture of \fig{cc-wc-performance-knn} repeats in \fig{cc-wc-performance-knn-log}: There are only a few cases that favor \textit{within} data and in the majority of the cases \textit{within} and \textit{cross} data have comparable performances.


\subsection{Selection Tendency}

To explore  selection tendency,
LOOCV is used to select out single test instances one by one from a \textit{within} dataset of size $T$.
The remaining $T-1$ \textit{within} instances are combined with the \textit{cross} subsets.
\begin{figure}[!t]
\renewcommand{\baselinestretch}{0.8}
\scriptsize
\centering
\begin{tabular}{l | l | l | l | l }
\textbf{Test Set} & \textbf{Zone} & \textbf{From S1} & \textbf{From S2}& \textbf{From S3} \\[1ex]\hline\hline
\input{resultsFile}
\end{tabular}
\caption{The amount of instances selected from within and cross company datasets. In parenthesis the percentage of selected instances out of the actual within source dataset is given. The diagonal entries that are highlighted with gray are the within source selection amounts and percentages.}
\label{fig:cc-selection}
\end{figure}

\begin{figure*}[th!]
  \centering                
  \subfigure[Percentages]{\label{fig:teak-percentage}\includegraphics[width=0.5\textwidth]{lib/teakWCCCPercentages.pdf}}
  \subfigure[Percentiles]{\label{fig:teak-percentile}\includegraphics[width=0.28\textwidth]{lib/teakWCCCPercentiles.pdf}}
  \caption{Percentages and percentiles of instances selected by TEAK from WC and CC datasets. The CC percentages are very similar to shifted version of WC percentages, the shift-effect is due to different number of subsets. The percentile graph removes the shift-effect and we see that \textit{within} test instances select very close percentages of \textit{within} and \textit{cross} source instances.}
  \label{fig:teak-perc}
\end{figure*}
Prior to combination, every training instance is marked with the
source that it belongs to (cross vs within).  Then the test instance
is allowed to choose \textit{k} analogies from a training set of
\textit{within} and \textit{cross} data.  

Thi rig lets us check
the percentage
selection of analogies from each one of the \textit{within} and
\textit{cross} subsets.  
Those results are shown in \fig{cc-selection}
\footnote{Ekrem- what is the
point of ``zone''- can we delete it?}.
Each
\textit{cross-within division} is represented with a row of $2$ or
$3$ subsets; columns named \textit{``From $S_i$''} where $i\in\{1,2,3\}$
represent the subsets of the rows:
\bi
\item
  The highlighted diagonal entries
of each cell show the amount of instances
selected from \textit{within} subset.  
\item
The off-diagonal
values are the amount of instances selected from \textit{cross}
datasets.
\ei
Note that  the \fig{cc-selection} results are expressed as raw values
as well (in brackets) the raw values divided by the number of instances
in the training set\footnote{ekrem- yes?}.
In order to determine the probability of selecting an instance
from {\em within} or {\em cross},
we compare the distribution of the diagonal 
and the off-diagonal elements. 

To better see the percentages of \textit{within} and \textit{cross}
subsets, we sorted and plotted them
in
\fig{teak-perc}.  \fig{teak-percentage} shows the sorted percentage
values, where the $within$ percentages are shown with circles, whereas
the $cross$ percentages are represented by rectangles\footnote{ekrem:
change key from wc cc to within cross}.  
Observe how 
the \textit{cross} percentage values are
shifted versions of \textit{within} percentages  
(this shift-effect
comes from the fact that there are more \textit{cross} subsets than
\textit{within} subsets).  

The percentiles from  $10^{th}$ to
$90^{th}$ with increments of $20$ are given in \fig{teak-percentile}.
When we plot the percentiles, the shift-effect due to subset number
disappears and we are able to observe that within
and cross selection tendancies
at the indicated percentile values are very close.
A statistical test (MANN-WHITNET, 95\% confidence)
confirms this: the distributions of 
\fig{teak-perc} are {\em
not} statistically significantly different.


%The percentage ordering plots of ABE0 and log+ABE0 are very similar to \fig{teak-percentage}.
%Therefore, we did not include these figures here due to space constraints.
%Instead, we give the percentile plots in \fig{percentile-abe0}.
%See in \fig{percentile-abe0} that percentage values of WC and CC are very close to one another for all percentile values.
%This means that subject to ABE methods with or without relevancy filtering, test instances select close amounts of WC and CC data.

\section{Discussion}
\subsection{Small Selection Sizes}
Initially, we were 
surprising about how $few$ instances
were selected 
by TEAC.
The median values of the percentiles
(i.e. $50^{th}$ percentile) in \fig{teak-perc}
is 7\%.
Initially, this low value troubled us but a review
of the instance selection literature found that this
was not an anomalous result.
For example,
Chang's prototype generators~\cite{chang74} replaced training sets
of size \mbox{$T=(514,150,66)$} with prototypes of size \mbox{$N=(34,14,6)$}  (respectively).
That is,  prototypes may be as few as $\frac{N}{T}=(7,9,9)\%$ of the original
data. 
\subsection{Geometry of Instance Space}
What do these results say about the
distribution of project data in instance space?
We argue that,
fundamentally, $locality(1)$ is an assumption about the position of
examples within instance space. Consider the circles of \fig{dim}-
these are projects described in terms of a two dimensional instance
space (labelled here as $x$ and $y$).  Note that (a) the circles are
arranged (approximately) parallel to the $y$ axis and that (b) the
longer projects (indicated with larger circles) occur at higher $y$
values. The space of the circle examples in \fig{dim} could be
processed by $locality(1)$ since a single feature (in this case, the
vertical $y-axs$) usefully divides the examples with higher effort
from those with lower effort.
\begin{figure}[!h]
\begin{minipage}{1in}
\includegraphics[width=1in]{xy}
\end{minipage}~~~~
\begin{minipage}{1.6in}
{\em
Squares and circles show examples from two different projects.

The size of each space  shows the effort required for that project.
}
\end{minipage}
\caption{Projects described in two dimensions.}\label{fig:dim}
\end{figure}

Now consider the squares in \fig{dim}. As before, the size
of the shapes  indicates the effort associated with each project.
Note these examples do not parallel to either feature and the
longer and shorter projects are not easily seperated by either axis.
The space of square circles require $locality(N)$: CART would need to
build multiple branches to capture the different regions of differing
effort; $k$-th nearest neighbor would need to use all features
and  $k=1$ in order to avoid learning from conflicting
examples (e.g. not two neighboring examples with very different efforts).


The results of this paper can be summarized as follows.
Given the idiosyncracies of software development, we find it
unlikely that naturally occuring project examples will all line
up in a row parallel to one axis. Hence,
$locality(N)$ is  be required
to divide complex spaces into local regions of neighboring similar properties.

%
%\begin{figure}[th!]
%  \centering                
%  \subfigure[ABE0 with \textit{k=4}]{\label{fig:4k-percentile}\includegraphics[width=0.23\textwidth]{lib/4kWCCCPercentiles.pdf}}
%  \subfigure[log+ABE0 with \textit{k=4}]{\label{fig:log4k-percentile}\includegraphics[width=0.23\textwidth]{lib/log4kWCCCPercentiles.pdf}}\\
%  \subfigure[ABE0 with \textit{k=best}]{\label{fig:bestk-percentile}\includegraphics[width=0.23\textwidth]{lib/bestkWCCCPercentiles.pdf}}
%  \subfigure[log+ABE0 with \textit{k=best}]{\label{fig:logbestk-percentile}\includegraphics[width=0.23\textwidth]{lib/logbestkWCCCPercentiles.pdf}}\\
%  \caption{Percentiles of instances selected by ABE0 and log+ABE0 with \textit{k=\{4, best\}} from WC and CC datasets.}
%  \label{fig:percentile-abe0}
%\end{figure}
%

\section{Threats to Validity}
\label{sec:threats-to-validity}
{\em External validity} questions whether the results can be generalized outside
the specifications of a study\cite{Milic2004}.  
For the purpose of external validity, we use of 21 WC-CC dataset pairs.
Among 10 studies investigated by Kitchenham et al. in~\cite{Kitchenham2007}, 9 of them used single WC-CC dataset pairs, and 1 study used 6 pairs.
In terms of external validity, this report has higher validity than a standard \textit{within} vs. \textit{cross} data comparison effort estimation study.

Another consideration for external validity is the employed methods.
Due to the nature of our research questions, particularly regarding selection tendency, we utilized ABE methods.
There are thousands of possible ABE variants and there is no way that this study covers them all.
There is obviously need for future research that repeats these experimentations with different ABE variants.
However, experiments reported here include base variants (ABE0 and log+ABE0) as well as filtering based variants (TEAK) run on 21 WC-CC pairs.
Therefore, the extent of the experimentation in this research offers enough support for the claims that 1) \textit{cross} data performs no worse than \textit{within} data and 2) a \textit{within} test instance tends to create an equal mixture of \textit{within} and \textit{cross} projects. 


{\em Construct validity} (i.e. face validity) asks if we are measuring
what we actually intended to measure~\cite{Robson2002}.
Previous studies have concerned themselves with the construct validity of 
different performance measures for effort estimation (e.g. \cite{Stensrud2002}).
So as not to bias our conclusions due to a limited number of performance measures, we used $4$ different performance measures aided with win-tie-loss statistics.
Another threat to construct validity is the formation of our \textit{cross-within divisions}.
The proposed division criteria for \textit{within} and \textit{cross} data may as well define different corporations in real world.
Also the use of available public data promotes the reproducibility of the conclusions of a research.
However, we acknowledge that using \textit{within} and \textit{cross} data coming from completely different organizations would be a better option.
On the other hand, the issue with using such proprietary data is the fact that they are difficult to be shared with research community and therefore proposed results cannot be reproduced, validated or refuted. 

In terms of {\em internal validity} of our results, there is one dimension of experimental conditions not explored.
We are making use of LOOCV, whose a possible alternative would be N-Way cross-validation.
In N-Way cross-validation, data is randomly divided into $B$ bins and each  bin is tested on a model learned from the combination of other bins (typical values for $B$ are 3 or 10).
From a theoretical point of view, not controlling the stability of our results across different testing strategies is a threat to validity, as different testing strategies entails different bias and variance conditions~\cite{Hastie2008}.
Elsewhere~\cite{Kocaguneli2011a}, we show that there is very little difference in the bias and variance values generated for LOOCV and N-way cross-validation. 
Since two testing strategies have similar bias-variance characteristics for effort datasets, we opted for LOOCV due to the fact that LOOCV is a deterministic procedure that can be exactly repeated by any other researcher with access to a particular data set.  
N-way cross-validation on the other hand requires a random number generator and a stratification heuristic (to maintain same class distribution in each bin).
Without access to exact same random number generator and stratification heuristic, it would be difficult for a researcher ``A'' to reproduce results of researcher ``B''.

\section{Conclusion}
We have shown that
when
using an effort estimator that is as good or better
than commonly used effort estimators (such as TEAC),
then if that estimator can select from $cross$ or $within$
soruces then:
\be
\item
The $cross$ performance results are no worse than $within$
(see \fig{cc-wc-performance-teak});
\item
The probability that the estimator selects a training instance
from $cross$ or $within$ is the same (see \fig{teac-perc}.b).
\ee


Result \#1 grants us permission to compare cross-vs-within results
(since there is no performance delta between them).
Result \#2 shows that the single-feature divisions have no
bearing on effort estimation.
Coupled with the results of~\cite{keung11},
these results are strong supportive for $locality(N)$ since we have
confirmed both {\bf PREDICTION~1} and {\bf PREDICTION~2}.

This means that (to  repeat a comment made in our introduction),
the most similar software to what you are writing now may not be in the next office. 
Rather, it may be in an office on the other side of the world.
As shown here,
using instance selection tools like TEAC, it is possible to automatically 
find that relevant training data.


As to our research questions:
%Conclusions: 1) Cross-company performance is comparable to within company data, 2) cross-company data is selected as much as the within company data, 3) current features that are known to define within company data do not apply..
%Only 5 to 10 \% matters...

\textbf{RQ1: What can be said about \textit{within} and \textit{cross}
data performances?} The results of TEAK experiments confirm the
findings of prior work~\cite{Turhan2009, Kocaguneli2010} on a larger
scale: After relevancy filtering \textit{cross} data performance
is comparable to that of \textit{within} data. 
 
\textbf{RQ2: What is the selection tendency of \textit{within} instances?}
A surprising result of this research is the fact that under different
ABE models (TEAK, ABE0, log+ABE0) test instances selected analogies
from \textit{within} as well as \textit{cross} data.  This may be
interpreted as a hint that organizations with very limited
\textit{within} data can incorporate \textit{cross} data into their
database.

%\textbf{RQ3: What is the reason for a particular tendency of \textit{within} instances?}
%In our experiments we have identified possible features in effort datasets for \textit{cross-within divisions}.
%Since test instances select analogies from all the subsets, the \textit{cross} subsets defined by these features are obviously good candidates to provide connections between \textit{cross} company data.
%
%\textbf{RQ4: What would be a good mixture of \textit{wihin} and \textit{cross} data?}
%The percentage distributions observed in \tion{results} show that the percentage amount of instances selected from \textit{within} and \textit{cross} datasets are very close to one another.
%So a good mixture seems to be equal percentages from different datasets, i.e. if an organization uses $x\%$ from their \textit{within} dataset, then the amount of instances selected from a \textit{cross} dataset could be close to $x\%$.
%Furthermore, the percentile figures show that median percentage value is between $5\%$ to $12\%$.
%This can be an indicator interval for the mixture of \textit{within} and \textit{cross} subsets.
%
%\textbf{RQ5: Under which conditions would \textit{cross} data be favorable?}
%Our results show that under 2 conditions \textit{cross} data would be beneficial for an organization: 1) When a relevancy filter is used and 2) when \textit{cross} data has a common feature (as those listed in this research) to the \textit{within} source.
%
\textbf{RQ3: Which features are likely to define \textit{cross} and \textit{within} boundaries?}
Our aim with this research question was to find features that would define boundaries to separate \textit{within} and \textit{cross} data from one another so that test instances would select mostly from its related \textit{within} dataset.
However, none of the features we have identified resulted in such a scenario.
On the contrary, they defined links between \textit{cross} datasets so that test instances selected from all subsets.

\section{Future Directions}
Some of the most likely future directions to this research are:
\bi
\item Reproduction of this work on proprietary data.
\item Investigating why particular subsets (cocomo81s, desharnaisL1) favor \textit{within} data, whereas the rest favors both \textit{within} and \textit{cross}.
\item Using different ABE or non-ABE methods under similar settings.
\item Experimenting if limited \textit{within} data can be supplemented with \textit{cross} data.
\item Using different features on different datasets to see if they can define a border between \textit{within} and \textit{cross} data.
\ei


\bibliographystyle{abbrv}
\bibliography{timm,library}

\balancecolumns
% That's all folks!
\end{document}


While it was not their intention to do so,
a recent ESEM paper on algorithmic methods
by Mair and Shepperd~\cite{Mair05}
comments on  {\bf PREDICTION~1}.
That paper searched the literature
for consistent results regarding the value
of regression or analogy-based methods for software effort estimation. 
If Mair and Shepperd had found that analogy always out-performs regressions, 
then that would have
been strong support for {\bf PREDICTION~1} since 
(a) regression builds one model across all variables;
and (b)
analogy-based methods  
use the $k$-thh nearest neighbors discussed in the introduction; i.e. they make estimates
using the subset of training data that clumps nearest the test instance.
However, their results were otherwise:
from a total of 20 empirical
studies, seven favored regression, four were indifferent and nine favored analogy.

Note that Mair and Shepperd did not re-run experiments: rather, they commented
on the consistency of empirical results generated by other researchers.
Keuing et al.~\cite{keung11} lament the state of empirical methods
in effort estimation. It is well established that any experiment ordering $M$ effort estimators
will generate different rankings, depending on the accuracy statistic used in the 
comparison; the random number generators used to build training and test sets;
and the data sets used~\cite{shepperd2003}. Unless these factors are controlled,
we must expect inconsistent results such as those reported
by Mair and Shepperd.

In an attempt to improve the state of empirical methods, Keung et al. explored the size
of the 
