\documentclass{acm_proc_article-sp}

\usepackage{times}
\usepackage{cite}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{multirow}
\usepackage{rotating}
\usepackage{subfigure}


%\usepackage[none]{hyphenat} 
\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{itemize}
  \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{itemize}}

\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
   \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}


\usepackage[table]{xcolor}
\usepackage{url,graphicx}
\newcommand{\G}{\cellcolor[rgb]{0.8,0.8,0.8}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\hyp}[1]{Hypothesis~\ref{hyp:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}

\newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
\newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}

\begin{document}

\title{Revisiting the merits of cross company data}


\numberofauthors{2} %  in this sample file, there are a *total*
% of EIGHT authors. SIX appear on the 'first-page' (for formatting
% reasons) and the remaining two appear in the \additionalauthors section.
%
\author{
% 1st. author
\alignauthor
Ekrem Kocaguneli\\
%	   \titlenote{Dr.~Trovato insisted his name be first.}\\
       \affaddr{Lane Department of Computer Science and Electrical Engineering}\\
       \affaddr{West Virginia University}\\
       \affaddr{Morgantown, USA}\\
       \email{ekocagun@mix.wvu.edu}
% 2nd. author
\alignauthor
Tim Menzies\\
%\titlenote{The secretary disavows
%any knowledge of this author's actions.}\\
       \affaddr{Lane Department of Computer Science and Electrical Engineering}\\
       \affaddr{West Virginia University}\\
       \affaddr{Morgantown, USA}\\
       \email{tim@menzies.us}
}
%\date{30 July 1999}
% Just remember to make sure that the TOTAL number of authors
% is the number that will appear on the first page PLUS the
% number that will appear in the \additionalauthors section.

\maketitle
\begin{abstract}
  ~\\\textit{\textbf{Background}}: Generating a database of past projects \textit{within} an organization requires considerable investment of 1) time, 2) money and 3) educated personnel.
Hence, it is tempting to \textit{cross} the boundaries of development type, location, language, application and hardware to use existing datasets of other organizations.
  \\\textit{\textbf{Aim}:} The literature is skeptical on the merits of cross-company data.
  Our hypothesis is that systematical investigation of project properties that are likely to define borders of \textit{crossness}, could reveal 1) how effective they can define these borders and 2) how much \textit{within} and \textit{cross} data would be favored by test instances.
  \\\textit{\textbf{Method}:} We filtered out 8 \textit{cross-within divisions} (21 pairs of \textit{within-cross} subsets) out of 19 datasets and evaluated these divisions under different analogy-based estimation (ABE) methods.
  \\\textit{\textbf{Results}:} We have seen that \textit{cross} and \textit{within} data is comparable in terms of: 1) performance subject to 4 evaluation criteria and 2) percentage instances selected by ABE methods in final estimation.
\end{abstract}


\category{H.4}{Software Cost Estimation}{\textit{k}-NN}
\category{D.2.8}{Software Engineering}{Cost}[within company, cross company]


\section{Introduction}

Accurate effort estimates of future projects are important for software organizations.
With precise estimates, the organizations can better allocate their resources; thereby, increasing their competitiveness on the market.
On the other hand, it is not easy to attain estimates with high accuracy values.
Even in corporations that have well established measurement and estimation practices, the matter of accuracy is questionable.
A well known example to that case is NASA's Check-out Launch Control System, that resulted in cancellation after doubling its initial estimate of \$200M \cite{CLCS03}.
The examples in less experienced corporations are even worse~\cite{Boehm2000}.

The accuracy of estimates depend on two fundamental factors: 1) the model used for estimations and 2) the historical database on which the model is built.
The second factor is somewhat a pre-condition of the first one, i.e. for a successful model, a well-maintained and precise dataset is a must.
Although both factors are equally important, here our focus will be more on the dataset. 

A company willing to undergo a project of a historical effort data collection \textit{within} the organization should be able to dedicate a considerable amount of time, money and personnel to this project.
Even if such an investment is ventured, the initial results may have to wait for a long time.
In the case of NASA-wide software cost metrics repository, only 7 projects could have been added in a time-frame of two years.
It is no surprise that a number of organizations cannot face the challenges associated with formation and maintenance of a \textit{within-company} (from now on WC) dataset.

An alternative approach to WC dataset formation is to adapt projects from \textit{cross} organizations.
With the help of publicly available data repositories, such cross-company (from now on CC) datasets are quite easy to find.
For example practitioners can find dozens of these datasets in PROMISE data repository~\cite{Boetticher2007}.

The merits of using CC data is contradictory.
A recent survey by Kitchenham et al. on the value of using CC data, shows that we are unable to make a conclusion~\cite{Kitchenham2007}.
Another study by Zimmermann et al. ends up with a similar result~\cite{Zimmermann2009}.
Turhan et al.~\cite{Turhan2009} and Kocaguneli et al.~\cite{Kocaguneli2010} show that through relevancy filtering CC data can perform as well as WC data.
Our work revisits the merits of \textit{cross} data by experimenting with various ABE methods (with and without relevancy filtering) on a larger scale.
We identified 21 WC-CC dataset pairs, which is orders of magnitude bigger than the amount of pairs used in previous studies~\cite{Kitchenham2007}.
Our conclusions from these experiments is that except for a very small minority of cases, CC data performance is comparable to WC.

Prior work has not deeply investigated the selection tendencies of \textit{within} company test instances.
We question how much data a test instance would select from \textit{within} and \textit{cross} data, when given access to both of them.
The results of this investigation are: 1) test instances tend to select analogies from both \textit{within} as well as \textit{cross} data and 2) percentage selections from WC and CC subsets tend to be very close to one another.

\subsection{Research Questions}
So as to guide this research the following research questions are defined:
\bi
\item[RQ1:] What can be said about \textit{within} and \textit{cross} data performances?
\item[RQ2:] What is the selection tendency of \textit{within} instances?
\item[RQ3:] What is the reason for a particular tendency of \textit{within} instances?
\item[RQ4:] What would be a good mixture of \textit{wihin} and \textit{cross} data?
\item[RQ5:] Under which conditions would \textit{cross} data be favorable?
\item[RQ6:] Which features are likely to define \textit{cross} and \textit{within} boundaries?
\ei

\section{Motivation}

There is enough empirical evidence that cross-company estimation is a serious problem to tackle\cite{Zimmermann2009, Turhan2009, Kitchenham2007, Kocaguneli2010}.
There are a still open questions to be further investigated and there is need for new questions to be raised. For example, the merits of using \textit{within} data is known~\cite{Zimmermann2009, Kitchenham2007}, i.e. we know that the model of \fig{within-problem} works successfully.
On the other hand, merits of \textit{cross} data in effort estimation is an open question.
The research done on \textit{cross} data so far is reported to be inconclusive~\cite{Kitchenham2007}, so we have both positive as well as negative results for the model of \fig{cross-problem}. 
There is still need for a comprehensive study on this issue.
In this study we investigate $21$ WC-CC pairs and compare their performances.

It has been shown that performance of \textit{cross} data can be improved via a relevancy filtering~\cite{Turhan2009, Kocaguneli2010}.
So it is known that model of \fig{cross-filter-problem} is a good candidate for making use of \textit{cross} data in software effort estimation.
In this work we use a variance-based relevancy filter in various scenarios.

The tendency of a test instance to select \textit{within} or \textit{cross} data instances has not been yet addressed.
The outcomes of the model in \fig{within-cross-problem} are not known.
The information about the selection tendency of a test instance can tell, if it is a good idea for an organization to combine limited \textit{within} data with \textit{cross} data.
It can also specify in which proportions such a mixture should be.
We define settings for that problem in this paper and look for answers to these problems.

\begin{figure}[th!]
  \centering                
  \subfigure[\textit{Within}]{\label{fig:within-problem}\includegraphics[width=0.09\textwidth]{withinProblem.png}}
  \subfigure[\textit{Cross}]{\label{fig:cross-problem}\includegraphics[width=0.09\textwidth]{crossProblem.png}} 
  \subfigure[\textit{Filter}]{\label{fig:cross-filter-problem}\includegraphics[width=0.09\textwidth]{crossFilterProblem.png}} 
  \subfigure[\textit{Within+Cross}]{\label{fig:within-cross-problem}\includegraphics[width=0.18\textwidth]{withinCrossProblem.png}} 
  \caption{The problem types in \textit{within} vs. \textit{cross} data comparison and our conclusions so far. \textit{``+''} and \textit{``-''} signs on top of models mean positive and negative results respectively. A \textit{``?''} sign means the model has not yet been investigated.}
  \label{fig:problem-types}
\end{figure}


\subsection{Contributions}
The contributions of this research can be listed as follows:
\bi
\item Analysis of effort datasets and proposition of subsets for possible WC-CC experiments.
\item Investigating effort dataset features  to see which features are likely to define \textit{cross} and \textit{within} boundaries.
\item An extensive analysis on the merits of \textit{cross} data and uniform conclusions.
\item Proposing a mixture model of \textit{within} and \textit{cross} data.
\item Investigating tendency of test instances towards \textit{cross} and \textit{within} data to find preferable percentage of a mixture.
\ei


\section{Related Work}

We divide our related work into two sub-sections.
\tion{effort-estimation} describes effort estimation in general and  \tion{cross-within} provides in depth discussion of \textit{cross} and \textit{within} data usage in SE.

\subsection{Effort Estimation}
\label{sec:effort-estimation}

A high level taxonomy of the software effort estimation based on the adopted methodology reveals two fundamental groups: Algorithmic and non-algorithmic methods. 

Effort estimates generated by algorithmic methods are the product of a model that is built on historical data.
Such methods may entail the adaptation of an expert proposed model to historical data.
A very well known example to that scenario is Boehm's COCOMO method~\cite{Boehm1981}.
However, this option requires a long time interval for \textit{within} data collection and model adaptation to local data.
An alternative is the processing of historical data by generic methods.
Regression methods~\cite{Menzies2006}, neural nets~\cite{Kultur2008}, model trees~\cite{Stewart2002} and instance-based models~\cite{Menzies2006, Kocaguneli2010, Kadoda2000} are examples to this category.

Another proposed effort estimation strategy is the non-algorithmic methods, a.k.a. expert-based estimation.
The models under this category can be defined as a human-intensive process of estimate negotiation between domain experts~\cite{Jorgensen2004}.
These negotiations continue until a concensus is reached among the experts.
Possible pitfalls related with this family of methods are threefold: 
\be
\item obvious need for high-quality experts, 
\item poor cabability of humans to improve their expert judgment skills~\cite{Jorgensen2009}  and 
\item possible conflict of interest among domain experts~\cite{Kocaguneli2010}.
\ee

\subsection{Within-Company vs. Cross-Company}
\label{sec:cross-within}

A baseline for successful algorithmic models is the historical effort datasets of past projects.
An organization willing to employ an algorithmic model-based effort estimation in their processes may choose to benefit from one of the following: 
\bi
\item \textit{within} data that is required to be collected \textit{within} the organization 
\item \textit{cross} data that was collected elsewhere and that needs to be adapted
\item \textit{combination} of \textit{within} and \textit{cross} data
\ei

The merits of \textit{within} data has been shown in the literature; for accurate estimates previous work suggest \textit{within} data that bears locality-speicific features.
However, there are multiple issues associated with collecting and maintaining \textit{within} data~\cite{Mendes2004, Mendes2005, Kitchenham2007, Kocaguneli2010}:
\be
\item Long time requirement for accumulation of enough local data
\item Possibility of technology change by the time \textit{within} data is ready
\item Sensitivity to possible human errors in data collection
\item Loss of managerial interest due to long time constraint
\ee

Unlike \textit{within} data, the reported results regarding \textit{cross} data are inconclusive.
An extensive systematic review by Kitchenham et al. on the value \textit{cross} data reports that  only 7 out of 10 studies in the review are able to show independent evidence in their comparisons of  \textit{within} and \textit{cross} data~\cite{Kitchenham2007}.
Out of this 7, 4 studies favor \textit{within} data, whereas the remaining 3 report that \textit{cross} data performance is not significantly worse than \textit{within}. 

Another field of SE that questions the merits of \textit{cross} data is defect prediction and the inconclusive scenario endures. 
Zimmermann et al.~\cite{Zimmermann2009} study \textit{cross} data in an as is manner and out of 622 \textit{cross} predictions only $3.4\%$ is reported to work.
Turhan et al. reports successful applications of \textit{cross} data: Filtering \textit{cross} data through a nearest-neighbor (NN) based filter increases defect prediction probability~\cite{Turhan2009}.
This filtering approach has also inspired effort estimation domain.
A variance based filtering on \textit{cross} data has attained comparable performance to that of \textit{within} data~\cite{Kocaguneli2010}.
The problem of \textit{cross} data has been paid little attention~\cite{Zimmermann2009} and new questions like the mixture of \textit{within} and \textit{cross} data) are yet to be raised.
Various aspects of this mixture approach are investigated in this paper.


\section{Methodology}


The details of dataset filtering and division of them into \textit{cross-within divisions} are presented here.
We also explain the selected performance measures and our experimental set-up.

\subsection{Datasets}
There are 2 fundamental factors that were considered for selection of the datasets used in this research:
\bi
\item Public availability: For reproducibility purposes
\item Cross-within divisibility: For enabling \textit{cross} vs. \textit{within} experimentation
\ei
A critical issue in software engineering is the ability of the proposed results to be reproducible~\cite{Juristo2009, Kitchenham2007} and use of proprietary data is a major obstacle towards this goal.
Therefore, all our datasets are publicly available through PROMISE data repository~\cite{Boetticher2007}.
Although there are more than twenty effort datasets available in PROMISE, they are not all available for \textit{cross-within} experimentation.
The available datasets should be able to provide \textit{cross-within division}.
We define \textit{cross-within division} as the subset(s) of effort data that are formed through division of a nominal attribute: Instances having the same value for that nominal attribute form a subset.
The nominal attribute should be a plausible candidate for a \textit{cross} company setting, i.e. the attribute should be likely to change from one company to other.

After manually inspecting more than 20 datasets and 6 are selected for \textit{cross-within} experimentation.
6 datasets defined 8 \textit{cross-within divisions} according to their available nominal attributes.
8 divisions include 21 subsets, i.e. 21 different WC-CC pairs.
The datasets, \textit{cross-within divisions} (subsets) and the division criteria are given in \fig{cc-selection}. 
The selected division criteria include:
\bi
\item project type: embedded, organic and semidetached (\textit{cocomo81}), 
\item center: geographical development center (\textit{nasa93}), 
\item language type: programming language used for development (\textit{desharnais}), 
\item application type: on-line service program, production control program etc. (\textit{finnish} and \textit{maxwell}), 
\item hardware: PC, mainframe, networked etc. (\textit{kemerer} and maxwell), 
\item source: whether in-house or outsourced (maxwell).
\ei

Note that each subset in \fig{cc-selection} is named with a self-explanatory abbreviation.
The numbers at the end of abbreviations correspond to values of the nominal attribute used to form the subsets.
If a name has multiple numbers at the end (e.g. finnishAppType2345) this means that all instances with these nominal attribute values are combined in a single subset.
Also note that the terms \textit{within/cross} and WC/CC will be used interchangeably for the rest of the text.

\begin{figure}
\scriptsize
\center
\begin{tabular}{l | l | l | r}
\textbf{Dataset} & \textbf{Criterion} & \textbf{Subsets} & \textbf{Subsets Size}\\[1ex]\hline\hline
cocomo81 & project type & cocomo81e &	28  \\
  & & cocomo81o   & 	24  \\
  & & cocomo81s   & 	11	  \\\hline
  nasa93 & development center & nasa93\_center\_1   & 	12  \\
  & & nasa93\_center\_2   & 	37  \\
  & & nasa93\_center\_5   & 	39  \\\hline	
  desharnais & language type & desharnaisL1   & 	46  \\
  & & desharnaisL2   & 	25  \\
  & & desharnaisL3   & 	10  \\\hline
finnish & application type & finnishAppType1 & 17 \\
 &  & finnishAppType2345 & 18 \\\hline
kemerer & hardware & kemererHardware1 & 7 \\
 &  & kemererHardware23456 & 8 \\\hline
maxwell & application type & maxwellAppType1 & 10 \\
 &  & maxwellAppType2 & 29 \\
 &  & maxwellAppType3 & 18 \\\hline
maxwell & hardware & maxwellHardware2 & 37 \\
 &  & maxwellHardware3 & 16 \\
 &  & maxwellHardware5 & 7 \\\hline
maxwell & source & maxwellSource1 & 8 \\
 &  & maxwellSource2 & 54 \\
\end{tabular}\caption{
6 datasets are selected from 20+ candidates.
Then selected datasets are divided into subsets according to a criterion that can define a \textit{cross-within division}. The datasets, subset sizes as well as the selection criteria are provided here.}
\label{fig:cc-selection}
\end{figure}

We acknowledge that \textit{cross-within division} is a validity concern (a detailed discussion is given in \tion{threats-to-validity}).
However, we hasten to say that it is an acceptable experimentation method, because: a) Despite all our efforts, public effort datasets are still limited; b) For a comprehensive analysis (as this study does on 21 subsets), alternative use of available datasets should be considered.

\subsection{Methods}
We used 2 different ABE methodologies in this paper: a relevancy filtering-based ABE method called ``Test Essential Assumption Knowledge'' (TEAK)~\cite{Kocaguneli2011} and ABE0~\cite{Kocaguneli2010,Kocaguneli2011}.

\subsubsection{ABE0: A Baseline Analogy-based Estimation}

ABE methods generate an estimate for a \textit{test} project by retrieving similar past projects (a.k.a. analogies) from a database of past projects and adapting their effort values into an estimate.
There are various design options associated with ABE methods such as the distance measure for nearness~\cite{Mendes2003}, adaptation of analogy effort values~\cite{Mendes2003}, row processing~\cite{Chang1974, Keung2008a}, column processing~\cite{Keung2008a, Li2009} and so on.
Elsewhere~\cite{Keung2011} we show that these options can easily lead to more than 6000 ABE variants. 
When ABE methods proposed by Kadoda \&
Shepperd~\cite{Kadoda2000},  Mendes et al.~\cite{Mendes2003}, and Li et al.~\cite{Li2009} are followed, a baseline variant emerges:
\bi
\item Form a database of past projects, whose rows are projects instances and whose columns are \textit{independent} variables (that define projects) and a \textit{dependent} variable (effort value). 
\item Decide how many similar projects (\textit{analogies}) are to be used from the training set, i.e \textit{k}-value.
\item For each test instance, retrieve \textit{k} analogies from the database.
\bi
\item For selection of \textit{k} analogies use a similarity measure like Euclidean distance measure.
\item Before calculating similarity, scale independent features to equalize their influence on the similarity measure.
\item Use a feature weighting scheme to reduce the effect of less informative features.
\ei
\item Adapt the effort values of the \textit{k} nearest analogies to come up with the effort estimate.
\ei
Following the steps of this baseline technique, we define a framework called ABE0.
ABE0 uses the Euclidean distance as a similarity measure, whose formula is given in Equation \ref{equ:euclid}, where $w_i$ corresponds to feature weights applied on independent features.
ABE0 framework does not favor any features over the others, therefore each feature has equal importance in ABE0, i.e. $w_i=1$.
For adaptation ABE0 takes the median of selected \textit{k} projects.

\begin{equation}\label{equ:euclid}
Distance = \sqrt{\sum_{i=1}^{n}{w_i(x_{i} - y_{i})^{2}}}
\end{equation}

\begin{figure}
\begin{center}
\includegraphics[width=0.35\textwidth]{gac-tree.png}
\end{center}
\caption{A sample GAC tree with regions of high variance (red) and low variance (green). GAC trees may not always be binary. For example here, leaves are odd numbered, hence node ``c'' is left \textit{behind}. Such instances are pushed \textit{forward} into the closest node in the higher level. For example, ``c'' is pushed forward into the ``b+f'' node to make ``b+f+c'' node.  
}\label{fig:gac-tree}
\end{figure}

\subsubsection{TEAK: Test Essential Assumption Knowledge}

TEAK is a relevancy filtering-based  ABE method that makes use of greedy-agglomerative clustering (GAC) trees.
Detailed description of TEAK can be found in~\cite{Kocaguneli2011}.
In summary, it is a two-pass system:
\bi
\item Pass 1 removes  the training instances implicated in poor decisions;
\item Pass 2 selects the instances that are closest to the test instance.
\ei

In the first pass, training instances are combined into a GAC tree (called GAC1).
A trivial example to GAC tree formation is provided in \fig{gac-tree}.
Level zero of GAC1 is formed by leaves, which are the individual project instances.
These instances are greedily combined into tuples to form the nodes of upper levels.
GAC1 is then traversed upwards from the root to level one (one level higher than the leaves).  
The variance of the effort values associated with each sub-tree (the performance variance)
is then recorded and normalized to a 0-1 interval.  
The high variance sub-trees are then chopped-off, as these are the sub-trees that would cause an ABE method to make an estimate from a highly variable instance space. 
Hence, pass one prunes sub-trees with a variance greater than $\alpha\%$
of the maximum variance seen in any tree.  After some experimentation,
we found that $\alpha=10$ lead to estimates with lowest errors.

The leaves of the remaining sub-trees are the {\em survivors} of pass one.
They are filtered to pass 2 where they are used to build a second GAC tree (GAC2).
GAC2 is generated and traversed in a similar fashion to GAC1, then test instances are moved from root to leaves.
Unlike GAC1, this time variance is a decision criterion for the movement of test instances: If the variance of the current tree is larger than its sub-trees, then continue to move down; otherwise, stop and select the instances in the current tree as the analogies.  
TEAK is a form of ABE0, so its adaptation method is the same, i.e. take the median of the analogy effort values.
A simple
visualization of this approach is given in in
\fig{teak}.

\begin{figure}
\begin{center}
\includegraphics[width=0.35\textwidth]{teak.png}
\end{center}
\caption{Execution of TEAK on 2 GAC trees, where tree on the left is GAC1 and the one on the left is GAC2 (i.e. lower variance sub-tree of GAC1). The instances in the low variance region of GAC1 (green region) are selected to form GAC2. Then test instance traverses GAC2 until no decrease in effort variance is possible. Wherever the test instance stops is selected as the subtree to be used for adaptation (white region of GAC2).   
}\label{fig:teak}
\end{figure}

\begin{figure}[!th]
\renewcommand{\baselinestretch}{.5}
\centering
\scriptsize
  \begin{tabular}{lccccccccc}
\multicolumn{10}{c}{20 $\times$ LEAVE-ONE-OUT }\\~\\~\\
&	\begin{sideways}\parbox{9mm}{TEAK}\end{sideways}	&	\begin{sideways}\parbox{9mm}{LR}\end{sideways}	&	\begin{sideways}\parbox{9mm}{NNet}\end{sideways}	&	\begin{sideways}\parbox{9mm}{\textit{k=best}}\end{sideways}	&	\begin{sideways}\parbox{9mm}{k=1}\end{sideways}	&	\begin{sideways}\parbox{9mm}{k=16}\end{sideways}	&	\begin{sideways}\parbox{9mm}{k=2}\end{sideways}	&	\begin{sideways}\parbox{9mm}{k=4}\end{sideways}	&	\begin{sideways}\parbox{9mm}{k=8}\end{sideways}	\\\hline
\multicolumn{10}{l}{  \underline{{\bf MdMRE}}}	\\
Cocomo81	   &	$\blacktriangle$ 	&		&		&	&		&		&		&		&		\\
Cocomo81e	    &	$\blacktriangle$ 	&		&		&		&		&		&		&		&		\\
Cocomo81o	    &	$\blacktriangle$ 	&		&		&		&		&		&		&		&		\\
Nasa93	        &	    &$\blacktriangle$ 	&	&		&		&		&		&		&		\\
Nasa93c2	    &	     &$\blacktriangle$	&	&		&		&		&		&		&		\\
Nasa93c5     	&	$\blacktriangle$ 	 &		&	&		&		&		&		&		&		\\
Desharnais	    &		     &$\blacktriangle$	&		&		&		&		&		&		&		\\
Sdr	            &	$\blacktriangle$	  &		&		&		& 	&		&		&		&		\\
ISBSG-Banking	&	$\blacktriangle$ 	 &		&		&		&		&		&		&		&		\\
\em Count	&	6	&	3	 &	0	&	0	&	0	&	0	&	0	&	0	&	0	\\


\hline\multicolumn{10}{l}{  \underline{{\bf Pred(25)}}}	\\
Cocomo81	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Cocomo81e	&		&		&	$\blacktriangle$	&		&		&		&		&		&		\\
Cocomo81o	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Nasa93	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Nasa93c2	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Nasa93c5	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Desharnais	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Sdr	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
ISBSG-Banking	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
\em Count	&	5	&	3	&	1	&	0	&	0	&	0	&	0	&	0	&	0	\\


\hline\multicolumn{10}{l}{ \underline{{\bf MAR}}}	\\
Cocomo81	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Cocomo81e	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Cocomo81o	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Nasa93	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Nasa93c2	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Nasa93c5	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
Desharnais	&		&	$\blacktriangle$	&		&		&		&		&		&		&		\\
Sdr	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
ISBSG-Banking	&	$\blacktriangle$	&		&		&		&		&		&		&		&		\\
\em Count	&	6	&	3	&	0	&	0	&	0	&	0	&	0	&	0	&	0	\\
\hline
\end{tabular}
\caption{
This figure displays the top  performing estimation methods,
measured via  $(win-loss)$ and repeated for the performance measures of MdMRE, Pred(25) and MAR.
The last row of each table shows the sum of times a method appeared as the top performing variant.
Note that TEAK is comparable to or better than non-analogy methods.
}
\label{fig:abe-vs-non-abe}
\end{figure}


\subsubsection{Why ABE methods: ABE0 and TEAK?}

The comparison of ABE methods vs. non-ABE methods (e.g. \textit{k}-NN vs. neural nets) is \textit{not} within the scope of this paper.
A detailed comparison of different methods on effort estimation can be found in~\cite{Menzies2006}.
The reasons behind selection of ABE methods (ABE0 and TEAK) in this research are threefold: 1) they are widely investigated in software effort estimation~\cite{Mendes2003, Chang1974, Keung2008a, Li2009, Kadoda2000,Kocaguneli2010,Kocaguneli2011}, 2) they are particularly helpful for \textit{cross} company examinations as they are based on distances between individual project instances and 3) analogy methods are comparable -if not better- to non-analogy methods in terms of performance.

In~\cite{Kocaguneli2011} we have compared performance of TEAK and ABE0 to non-analogy estimators (neural networks (NNet) and linear regression (LR)).
An excerpt from that comparison is given in \fig{abe-vs-non-abe} (for a complete analysis and for definitions of datasets please refer to Figure 7 of~\cite{Kocaguneli2011}).
Note in \fig{abe-vs-non-abe} that TEAK is usually the high performer in comparison to non-ABE methods.


\subsection{Performance Measures}
\label{sec:performance-measures}
A performance measure comments on the success of an estimate, hence the predictor.
Performance measures listed here have the property that there is at least one publication in effort estimation research proposing their use.
An example performance measure is the absolute residual (AR), which is the absolute difference between predicted and the actual effort values.
AR formula is given in \eq{ar}, where $x_i,\hat{x_i}$ are the actual and predicted values respectively for test instance $i$.
Summary of individual AR values is found by taking their mean (MAR).

{\begin{equation}\label{eq:ar}
AR_i=|x_i-\hat{x_i}|\
\end{equation}}

Another performance measures is the Magnitude of Relative Error (MRE), which is a widely used method to select the best estimator from a number of competing models~\cite{Shepperd1997, Foss2003}. 
MRE is the measure of the error ratio between the actual and the predicted effort:  
{\begin{equation}\small\label{one}
MRE_i=\frac{\mid{x_i-\hat{x_i}}\mid}{{x_i}} = \frac{\mid{AR_i}\mid}{{x_i}}
\end{equation}}


MRE can be summarized through mean or median.
The former summary defines mean MRE (MMRE) and the latter defines median MRE (MdMRE).
Formulas of MMRE and MdMRE are: 


{\begin{eqnarray}\small\label{one}
MMRE = mean(MRE_1, MRE_2, ... , MRE_n ) \\
MdMRE = median(MRE_1, MRE_2, ... , MRE_n )
\end{eqnarray}}

An alternative to prior performance measures is $Pred(x)$, i.e. the percentage of estimates that fall within $x\%$ of the actual values:
{\begin{equation}\small
Pred(x) = \frac{100}{N}\sum_{i=1}^N\left\{\begin{array}{ll}
\mbox{1 if $MRE_i \le \frac{x}{100}$}\\
\mbox{0 otherwise}
\end{array}\right.
\end{equation} 
}
A common value for $x$ in $Pred(x)$ is $30$~\cite{Chulani1999}.
For example, $Pred(30)=50\%$ implies that half of the estimates are within $30\%$ of the actual values \cite{Shepperd1997}.


\begin{figure}
\renewcommand{\baselinestretch}{1}
\begin{algorithmic} \footnotesize
\STATE \textit{$win_{i}$} = 0, \textit{$tie_{i}$} = 0, \textit{$loss_{i}$} = 0
\STATE \textit{$win_{j}$} = 0, \textit{$tie_{j}$} = 0, \textit{$loss_{j}$} = 0
\IF{WILCOXON(\textit{$P_i$}, \textit{$P_j$}) says they are the same}
	\STATE \textit{$tie_{i}$} = \textit{$tie_{i}$} + 1;
	\STATE \textit{$tie_{j}$} = \textit{$tie_{j}$} + 1;
\ELSE
	\IF{mean or median(\textit{$P_{i}$}) $<$ median(\textit{$P_{j}$})}
		\STATE \textit{$win_{i}$} = \textit{$win_{i}$} + 1
		\STATE \textit{$loss_{j}$} = \textit{$loss_{j}$} + 1
	\ELSE
		\STATE \textit{$win_{j}$} = \textit{$win_{j}$} + 1
		\STATE \textit{$loss_{i}$} = \textit{$loss_{i}$} + 1
	\ENDIF
\ENDIF
\end{algorithmic}
\caption{Pseudocode for win-tie-loss calculation between methods \textit{i} and \textit{j}
w.r.t. performance measures $P_i$ and $P_j$. Note here that only for Pred(30) the comparison is based on actual values ($Pred(30)_i$, $Pred(30)_j$) rather than mean or median values of performance measure arrays (median(\textit{$P_{i}$}), median(\textit{$P_{j}$})).}
\label{fig:win-tie-loss-pseudo}
\end{figure}

It is reported as a considerable threat to the validity of effort estimation studies, if derived performance measures are not evaluated with an appropriate statistical test~\cite{Kitchenham2009}.
In this study we use a Mann Whitney test (95\%).
To compare the values of MAR, MMRE, MdMRE and Pred(30), we use the following win-tie-loss procedure that incorporates Mann Whitney statistical test.
We first check if  two distributions $i,j$ are statistically different according to the Mann Whitney test.  
In our experimental setting, $i,j$ are arrays of performance measure results coming from two different methods.
If they are not statistically different, then they are said to \textit{tie} and we increment $tie_{i}$ and $tie_{j}$.  
On the contrary, if they are different, we updated $win_{i},win_{j}$ 
and ${loss_{i},loss_j}$ after a numerical comparison of performance measures.
The related pseudocode is given in \fig{win-tie-loss-pseudo}.
To get rid of any bias that would come from a particular experimental setting, for every experiment $20$ runs are made.


\subsection{Experimentation}
The experimentation of this research has two different scenarios: Performance comparison and selection tendency.
Performance comparison scenario compares different ABE methods (TEAK and ABE0 with \textit{k={4,best}}) when subject to \textit{within} and \textit{cross} company data.
The selection tendency experiments question the tendency of a \textit{within} test instance towards \textit{within} or \textit{cross} data.
In other words, given the chance that a test instance had access to \textit{within} and \textit{cross} data at the same time, what percentage of every subset would be selected into \textit{k} analogies used for estimation.
The details of each scenario is presented in the following subsections.

\subsubsection{Performance Comparison}
For performance comparison scenario we have two settings: \textit{Within} and \textit{cross}.
In \textit{within} data setting, only \textit{within} data is used as the dataset and a testing strategy of leave-one-out cross-validation (LOOCV) is employed. 
LOOCV works as follows: Given a \textit{within} dataset of $T$ projects, $1$ project at a time is selected as the test  and the remaining $T-1$ projects are used for  training, so eventually we have $T$ predictions. The resulting $T$ predictions are then used to compute $4$ different performance measures of \tion{performance-measures}.

\textit{Cross} data setting uses \textit{within} data as the test set and the \textit{cross} data as the training set.
This setting is basically a simulation of an organization that has projects to estimate (\textit{within} data as the test set) and uses data of past projects with  recorded effort values from \textit{cross} organization(s).
In this setting LOOCV is used: Each \textit{within} project is selected as the test instance and ABE methods derive an estimate for that instance by adapting \textit{cross} analogies.
Ultimately we end up with $T$ predictions adapted from a \textit{cross} dataset.
Finally the performances of TEAK and ABE0 methods under \textit{within} and \textit{cross} data settings are compared.
For that purpose we use both mere performance values as well as win-tie-loss statistics.

\subsubsection{Selection Tendency}
For the selection tendency scenario we select test instances according to LOOCV.
For each test instance, we are left with training sets of $T-1$ \textit{within} data and the subsets of \textit{cross} data.
After marking every \textit{within} and \textit{cross} instance, we combine the two datasets into a single training set and let the test instance choose analogies from the unified training set.
In this setting our aim is to see what percentage of \textit{within} and \textit{cross} subsets would appear among selected \textit{k} analogies.
The percentage for a subset $S_i$ is calculated in accordance with \eq{percentage}:

\begin{equation}\label{eq:percentage}
Percentage = \frac{\# ofInstances From S_i In Analogies}{Size(S_i)}
\end{equation}

\section{Results}
\label{sec:results}

\subsection{Performance Comparison}

The first experimental scenario we are interested in is the performance of \textit{within} and \textit{cross} data.
For performance comparison 4 different performance measures are employed: MAR, MMRE, MdMRE and Pred(30).
The actual performance values are also evaluated subject to Mann Whitney statistical test at $95\%$ confidence and this evaluation is summarized by win-tie-loss statistics.
Performance comparison subject to estimates of TEAK, ABE0 and log+ABE0 is reported in \fig{cc-wc-performance-teak}, \fig{cc-wc-performance-knn} and \fig{cc-wc-performance-knn-log} respectively.

\fig{cc-wc-performance-teak} shows \textit{within} and \textit{cross} data performance when TEAK is used as the estimation method.
For each performance measure win-tie-loss statistics  (abbreviated with \textbf{W}, \textbf{T}, \textbf{L} respectively) of \textbf{WC} performance when compared to \textbf{CC} over 20 runs as well as actual performance measure values are reported.
For convenience, the cases where \textbf{WC} is \textit{``dominantly''} superior to \textbf{CC} are highlighted.
A \textit{``dominant''} superior condition means winning more than half the runs, i.e. a \textbf{W} value of more than 10.
Note that there are only 2 such cases, where \textbf{WC} is dominantly better than \textbf{CC}: cocomo81s and desharnaisL1.
For the remaining $19$ cases \textit{within} data does not provide an advantage over \textit{cross} data.
In one particular case (kemererHardware1) the \textit{within} data is far worse than \textit{cross} with an \textbf{L} value of 20.
These results are confirmation of previous conclusions~\cite{Turhan2009, Kocaguneli2010} in a much larger scale with $4$ error measures and $21$ different cases: Relevancy filtering on \textit{cross} data improves its performance to an extent where it is no worse than \textit{within} data.
One likely question to be raised is why particular cases favor \textit{within} or \textit{cross} data.
This question is out of our scope and is left as a future direction to this research.

\begin{figure}[th!]
\renewcommand{\baselinestretch}{0.8} 
\begin{sideways}
\scriptsize
\centering
\begin{tabular}{l | l l l | l l | l l l | l l | l l l | l l | l l l | l l }
\textbf{Dataset} & \multicolumn{5}{c |}{\textbf{MAR}} & \multicolumn{5}{c |}{\textbf{MMRE}} & \multicolumn{5}{c |}{\textbf{MdMRE}} & \multicolumn{5}{c}{\textbf{Pred(30)}}  \\\hline
        & \textbf{W} & \textbf{T} & \textbf{L}  & \textbf{WC}  & \textbf{CC} & \textbf{W} & \textbf{T} & \textbf{L}  & \textbf{WC}  & \textbf{CC} & \textbf{W} & \textbf{T} & \textbf{L}  & \textbf{WC}  & \textbf{CC} & \textbf{W} & \textbf{T} & \textbf{L}  & \textbf{WC}  & \textbf{CC} \\
\input{winTieLossFileAll}
\end{tabular}
\end{sideways}
\caption{Results of TEAK: Comparison of performance between \textit{within} and \textit{cross} data w.r.t. 4 different performance measures (MAR, MMRE, MdMRE, Pred(30)) as well as \textbf{\textit{W, T, L}} statistics. Highlighted rows are the cases, where \textit{within} data is \textit{``dominantly''} better than \textit{cross}, i.e. wins more than half the time. Under the columns of \textbf{\textit{WC}} and \textbf{\textit{CC}} the actual performance values associated with \textit{within} and \textit{cross} company datasets are provided respectively.}
\label{fig:cc-wc-performance-teak}
\end{figure}

Another intriguing question is what happens in this large-scale comparison, when we remove the relevancy filtering.
The performance comparison of \textit{within} and \textit{cross} data subject to estimates of ABE0 with \textit{k=\{4, best\}} is given in \fig{cc-wc-performance-knn}.
The cells of \fig{cc-wc-performance-knn} can have 3 values: \textit{``+'', ``-'' and ``o''.}. The \textit{``+''} sign tells that \textit{within} data performance is \textit{``dominantly''} better than that of \textit{cross} data, i.e. \textit{within} won more than 10 out of 20 runs, whereas a \textit{``-''} sign tells that \textit{within} lost more than 10 runs. If none of these conditions occur, i.e. \textit{within} and \textit{cross} performances tie, then a ``o'' sign is assigned to the cell.
The cases where \textit{within} data is \textit{``dominantly''} better than \textit{cross} are highlighted.

\begin{figure}[th!]\renewcommand{\baselinestretch}{0.8}
\scriptsize
\begin{tabular}{l | l l | l l | l l | l l }
\textbf{Dataset} & \multicolumn{2}{c |}{\textbf{MAR}} & \multicolumn{2}{c |}{\textbf{MMRE}} & \multicolumn{2}{c |}{\textbf{MdMRE}} & \multicolumn{2}{c}{\textbf{Pred(30)}}  \\\hline\hline
         & \begin{sideways}\textbf{\textit{k}=4}\end{sideways} & \begin{sideways}\textbf{\textit{k}=best}\end{sideways} 
         &  \begin{sideways}\textbf{\textit{k}=4}\end{sideways} & \begin{sideways}\textbf{\textit{k}=best}\end{sideways}  
         & \begin{sideways}\textbf{\textit{k}=4}\end{sideways} & \begin{sideways}\textbf{\textit{k}=best} \end{sideways}
         & \begin{sideways}\textbf{\textit{k}=4}\end{sideways} & \begin{sideways}\textbf{\textit{k}=best}\end{sideways} \\\hline
\input{winTieLossKNN}
\end{tabular}
\caption{ABE0 performance comparison between \textit{within} and \textit{cross} data w.r.t. 4 different performance measures (MAR, MMRE, MdMRE, Pred(30)) with different \textit{k} values. Each cell in this table can have three values: \textit{``+'', ``-'' and ``o''.}. A \textit{``+''} sign indicates that \textit{within} performance is \textit{``dominantly''} better than \textit{cross}, i.e. it won more 10 of the 20 runs, whereas a \textit{``-''} sign tells that \textit{cross} lost more than 10 runs. If none of these conditions occur, i.e. \textit{within} and \textit{cross} performances tie, then a ``o'' sign is assigned to the cell. For convenience \textit{``+''} signs are highlighted.}
\label{fig:cc-wc-performance-knn}
\end{figure}

Notice in \fig{cc-wc-performance-knn} that out of \textit{21 Subsets} $\times$ \textit{4 error measures} $\times$ \textit{2 ABE0 methods} = \textit{168 cases}, there are only $3$ cases where \textit{within} performance is dominantly better. 
For the majority of the remaining cases, \textit{within} and \textit{cross} performance is comparable.
Surprisingly for a minority of the remaining cases (see ``-'' signs), \textit{cross} data performance is dominantly better than that of \textit{within}.
These results are important in the sense that  unlike previous work that report inconclusive results on the merits of cross data~\cite{Mendes2007, Zimmermann2009}; we are able to see enough uniformity in a large scale experiment with $168$ cases.
The uniformity concludes that \textit{cross} data is as high performing as \textit{within} data.
However, we should remember that such uniformity may come from \textit{cross-within divisions}, which assume \textit{cross} subsets of a division are all collected through the same methodology.

\begin{figure}[th!]\renewcommand{\baselinestretch}{0.8}
\scriptsize
\begin{tabular}{l | l l | l l | l l | l l }
\textbf{Dataset} & \multicolumn{2}{c |}{\textbf{MAR}} & \multicolumn{2}{c |}{\textbf{MMRE}} & \multicolumn{2}{c |}{\textbf{MdMRE}} & \multicolumn{2}{c}{\textbf{Pred(30)}}  \\\hline\hline
         & \begin{sideways}\textbf{\textit{k}=4}\end{sideways} & \begin{sideways}\textbf{\textit{k}=best}\end{sideways} 
         &  \begin{sideways}\textbf{\textit{k}=4}\end{sideways} & \begin{sideways}\textbf{\textit{k}=best}\end{sideways}  
         & \begin{sideways}\textbf{\textit{k}=4}\end{sideways} & \begin{sideways}\textbf{\textit{k}=best} \end{sideways}
         & \begin{sideways}\textbf{\textit{k}=4}\end{sideways} & \begin{sideways}\textbf{\textit{k}=best}\end{sideways} \\\hline
\input{winTieLossKNNLog}
\end{tabular}
\caption{\textit{log}+ABE0 performance comparison between \textit{within} and \textit{cross} data w.r.t. 4 different performance measures (MAR, MMRE, MdMRE, Pred(30)) with different \textit{k} values. The notation used here is the same as \fig{cc-wc-performance-knn}. The highlighted cells are the cases, where \textit{within} data is dominantly better than \textit{cross} data.}
\label{fig:cc-wc-performance-knn-log}
\end{figure}

The pre-processors applied on the data before an estimation model can have a significant effect on the performance~\cite{Menzies2006}.
In~\cite{Keung2011} we investigate the stability of rankings among $90$ different methods that include both analogy and non-analogy methods.
We have seen that applying a \textit{log} transformation on the data as a pre-processing step to ABE0 (log+ABE0), can improve the ranking of ABE0 by orders of magnitude.
The results of log+ABE0 is summarized in \fig{cc-wc-performance-knn-log}.
The notation of \fig{cc-wc-performance-knn-log} is the same as that of \fig{cc-wc-performance-knn}.
See that the general picture of \fig{cc-wc-performance-knn} repeats in \fig{cc-wc-performance-knn-log}: There are only a few cases that favor \textit{within} data and in the majority of the cases \textit{within} and \textit{cross} data have comparable performances.

\subsection{Selection Tendency}

The second experimental scenario of this research is the selection tendency.
In this setting LOOCV is used to select out single test instances one by one from a \textit{within} dataset of size $T$.
Remaining $T-1$ \textit{within} instances are combined with the \textit{cross} subsets.

\begin{figure}[th!]
\renewcommand{\baselinestretch}{0.8}
\scriptsize
\centering
\begin{tabular}{l | l | l | l | l }
\textbf{Test Set} & \textbf{Zone} & \textbf{From S1} & \textbf{From S2}& \textbf{From S3} \\[1ex]\hline\hline
\input{resultsFile}
\end{tabular}
\caption{The amount of instances selected from within and cross company datasets. In parenthesis the percentage of selected instances out of the actual within company dataset is given. The diagonal entries that are highlighted with gray are the within company selection amounts and percentages.}
\label{fig:cc-selection}
\end{figure}

\begin{figure*}[th!]
  \centering                
  \subfigure[Percentages]{\label{fig:teak-percentage}\includegraphics[width=0.5\textwidth]{lib/teakWCCCPercentages.pdf}}
  \subfigure[Percentiles]{\label{fig:teak-percentile}\includegraphics[width=0.28\textwidth]{lib/teakWCCCPercentiles.pdf}}
  \caption{Percentages and percentiles of instances selected by TEAK from WC and CC datasets. The CC percentages are very similar to shifted version of WC percentages, the shift-effect is due to different number of subsets. The percentile graph removes the shift-effect and we see that \textit{within} test instances select very close percentages of \textit{within} and \textit{cross} company instances.}
  \label{fig:teak-perc}
\end{figure*}

Prior to combination, every training instance is marked with the source that it belongs to.
Then the test instance is allowed to choose \textit{k} analogies from a training set of \textit{within} and \textit{cross} data.
After processing test instances via TEAK, ABE0 and log+ABE0; we can see the percentage selection of analogies from each one of the \textit{within} and \textit{cross} subsets.
\fig{cc-selection} shows the size of prediction \textbf{\textit{``Zone''}} used by TEAK for estimation as well as the percentage selection of instances from \textit{within} and \textit{cross} subsets into this prediction zone.
Each \textit{cross-within division} is represented with a row of $2$ or $3$ subsets; columns named \textit{``From $S_i$''} where $i\in\{1,2,3\}$ represent the subsets of the rows.
The highlighted diagonal entries of each cell show the amount of instances (in percentage and in number) selected from \textit{within} subset. 
The off-diagonal values are the amount of instances selected from \textit{cross} datasets.
See in \fig{cc-selection} that \textit{within} test instances do not necessarily select all the analogies from \textit{within} subsets.
On the contrary, the percentage of instances selected from \textit{within} and \textit{cross} datasets are very close to one another. 

To better see the percentages of \textit{within} and \textit{cross} subsets, we sorted and plotted these percentage values in \fig{teak-perc}.
\fig{teak-percentage} shows the sorted percentage values, where the WC data percentages are shown with circles, whereas the CC data percentages are represented by rectangles.
See in \fig{teak-percentage} how \textit{cross} percentage values are shifted versions of \textit{within} percentages.
The shift-effect comes from the fact that there are more \textit{cross} subsets than \textit{within} subsets.
The percentiles from  $10^{th}$ to  $90^{th}$ with increments of $20$ are given in \fig{teak-percentile}.
When we plot the percentiles, the shift-effect due to subset number disappears and we are able to observe the surprising fact that WC and CC percentages at indicated percentile values are very close.
In other words, subject to a variance based relevancy filtering ABE model, a \textit{within} test instance selects equal percentages from \textit{within} and \textit{cross} datasets.

The percentage ordering plots of ABE0 and log+ABE0 are very similar to \fig{teak-percentage}.
Therefore, we did not include these figures here due to space constraints.
Instead, we give the percentile plots in \fig{percentile-abe0}.
See in \fig{percentile-abe0} that percentage values of WC and CC are very close to one another for all percentile values.
This means that subject to ABE methods with or without relevancy filtering, test instances select close amounts of WC and CC data.

We can now ask what percentage should be used when an organization has certain amount of \textit{within} data but also needs the use of a \textit{cross} dataset.
The median values of the percentiles (i.e. $50^{th}$ percentile) in \fig{percentile-abe0} lie somewhere between $5\%$ to $12\%$.
This number can be an indicator for a mixture amount.
However, we do not have enough evidence to claim such a value yet.
Obviously further research is required to propose such a mixture amount and on the relative performances of different mixtures.


\begin{figure}[th!]
  \centering                
  \subfigure[ABE0 with \textit{k=4}]{\label{fig:4k-percentile}\includegraphics[width=0.23\textwidth]{lib/4kWCCCPercentiles.pdf}}
  \subfigure[log+ABE0 with \textit{k=4}]{\label{fig:log4k-percentile}\includegraphics[width=0.23\textwidth]{lib/log4kWCCCPercentiles.pdf}}\\
  \subfigure[ABE0 with \textit{k=best}]{\label{fig:bestk-percentile}\includegraphics[width=0.23\textwidth]{lib/bestkWCCCPercentiles.pdf}}
  \subfigure[log+ABE0 with \textit{k=best}]{\label{fig:logbestk-percentile}\includegraphics[width=0.23\textwidth]{lib/logbestkWCCCPercentiles.pdf}}\\
  \caption{Percentiles of instances selected by ABE0 and log+ABE0 with \textit{k=\{4, best\}} from WC and CC datasets.}
  \label{fig:percentile-abe0}
\end{figure}


\section{Threats to Validity}
\label{sec:threats-to-validity}
{\em External validity} questions whether the results can be generalized outside
the specifications of a study\cite{Milic2004}.  
For the purpose of external validity, we use of 21 WC-CC dataset pairs.
Among 10 studies investigated by Kitchenham et al. in~\cite{Kitchenham2007}, 9 of them used single WC-CC dataset pairs, and 1 study used 6 pairs.
In terms of external validity, this report has higher validity than a standard \textit{within} vs. \textit{cross} data comparison effort estimation study.

Another consideration for external validity is the employed methods.
Due to the nature of our research questions, particularly regarding selection tendency, we utilized ABE methods.
There are thousands of possible ABE variants and there is no way that this study covers them all.
There is obviously need for future research that repeats these experimentations with different ABE variants.
However, experiments reported here include base variants (ABE0 and log+ABE0) as well as filtering based variants (TEAK) run on 21 WC-CC pairs.
Therefore, the extent of the experimentation in this research offers enough support for the claims that 1) \textit{cross} data performs no worse than \textit{within} data and 2) a \textit{within} test instance tends to create an equal mixture of \textit{within} and \textit{cross} projects. 


{\em Construct validity} (i.e. face validity) asks if we are measuring
what we actually intended to measure~\cite{Robson2002}.
Previous studies have concerned themselves with the construct validity of 
different performance measures for effort estimation (e.g. \cite{Stensrud2002}).
So as not to bias our conclusions due to a limited number of performance measures, we used $4$ different performance measures aided with win-tie-loss statistics.
Another threat to construct validity is the formation of our \textit{cross-within divisions}.
The proposed division criteria for \textit{within} and \textit{cross} data may as well define different corporations in real world.
Also the use of available public data promotes the reproducibility of the conclusions of a research.
However, we acknowledge that using \textit{within} and \textit{cross} data coming from completely different organizations would be a better option.
On the other hand, the issue with using such proprietary data is the fact that they are difficult to be shared with research community and therefore proposed results cannot be reproduced, validated or refuted. 

In terms of {\em internal validity} of our results, there is one dimension of experimental conditions not explored.
We are making use of LOOCV, whose a possible alternative would be N-Way cross-validation.
In N-Way cross-validation, data is randomly divided into $B$ bins and each  bin is tested on a model learned from the combination of other bins (typical values for $B$ are 3 or 10).
From a theoretical point of view, not controlling the stability of our results across different testing strategies is a threat to validity, as different testing strategies entails different bias and variance conditions~\cite{Hastie2008}.
Elsewhere~\cite{Kocaguneli2011a}, we show that there is very little difference in the bias and variance values generated for LOOCV and N-way cross-validation. 
Since two testing strategies have similar bias-variance characteristics for effort datasets, we opted for LOOCV due to the fact that LOOCV is a deterministic procedure that can be exactly repeated by any other researcher with access to a particular data set.  
N-way cross-validation on the other hand requires a random number generator and a stratification heuristic (to maintain same class distribution in each bin).
Without access to exact same random number generator and stratification heuristic, it would be difficult for a researcher ``A'' to reproduce results of researcher ``B''.

\section{Conclusion}

At the end of an extensive experimentation, we can answer the research questions that were defined to guide this research.
%Conclusions: 1) Cross-company performance is comparable to within company data, 2) cross-company data is selected as much as the within company data, 3) current features that are known to define within company data do not apply..
%Only 5 to 10 \% matters...

\textbf{RQ1: What can be said about \textit{within} and \textit{cross} data performances?}
The results of TEAK experiments confirm the findings of prior work~\cite{Turhan2009, Kocaguneli2010} on a larger scale: After relevancy filtering \textit{cross} data performance is comparable to that of \textit{within} data.
The results of ABE0 experiments are quite interesting.
For the selected effort datasets, there are very limited cases, where \textit{within} data  outperforms \textit{cross} data.
For the majority of the cases, \textit{within} and \textit{cross} data performances are comparable.
The possible explanation for this outcome may be hidden in the fact that subsets of a particular effort dataset share similar methodologies and similar concerns in the collection process.
However, we do not yet have enough evidence to strongly support that hypothesis.

\textbf{RQ2: What is the selection tendency of \textit{within} instances?}
A surprising result of this research is the fact that under different ABE models (TEAK, ABE0, log+ABE0) test instances selected analogies from \textit{within} as well as \textit{cross} data.
This may be interpreted as a hint that organizations with very limited \textit{within} data can incorporate \textit{cross} data into their database.

\textbf{RQ3: What is the reason for a particular tendency of \textit{within} instances?}
In our experiments we have identified possible features in effort datasets for \textit{cross-within divisions}.
Since test instances select analogies from all the subsets, the \textit{cross} subsets defined by these features are obviously good candidates to provide connections between \textit{cross} company data.

\textbf{RQ4: What would be a good mixture of \textit{wihin} and \textit{cross} data?}
The percentage distributions observed in \tion{results} show that the percentage amount of instances selected from \textit{within} and \textit{cross} datasets are very close to one another.
So a good mixture seems to be equal percentages from different datasets, i.e. if an organization uses $x\%$ from their \textit{within} dataset, then the amount of instances selected from a \textit{cross} dataset could be close to $x\%$.
Furthermore, the percentile figures show that median percentage value is between $5\%$ to $12\%$.
This can be an indicator interval for the mixture of \textit{within} and \textit{cross} subsets.

\textbf{RQ5: Under which conditions would \textit{cross} data be favorable?}
Our results show that under 2 conditions \textit{cross} data would be beneficial for an organization: 1) When a relevancy filter is used and 2) when \textit{cross} data has a common feature (as those listed in this research) to the \textit{within} company.

\textbf{RQ6: Which features are likely to define \textit{cross} and \textit{within} boundaries?}
Our aim with this research question was to find features that would define boundaries to separate \textit{within} and \textit{cross} data from one another so that test instances would select mostly from its related \textit{within} dataset.
However, none of the features we have identified resulted in such a scenario.
On the contrary, they defined links between \textit{cross} datasets so that test instances selected from all subsets.

\section{Future Directions}
Some of the most likely future directions to this research are:
\bi
\item Reproduction of this work on proprietary data.
\item Investigating why particular subsets (cocomo81s, desharnaisL1) favor \textit{within} data, whereas the rest favors both \textit{within} and \textit{cross}.
\item Using different ABE or non-ABE methods under similar settings.
\item Experimenting if limited \textit{within} data can be supplemented with \textit{cross} data.
\item Using different features on different datasets to see if they can define a border between \textit{within} and \textit{cross} data.
\ei


\bibliographystyle{abbrv}
\bibliography{library}

\balancecolumns
% That's all folks!
\end{document}
