\documentclass{acm_proc_article-sp}

\usepackage{graphicx} 
\usepackage{subfig}
\usepackage{verbatim}
\usepackage{fancyvrb}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{timtricks}
\usepackage{cite}

\begin{document}


%\CopyrightYear{2007} % Allows default copyright year (200X) to be over-ridden - IF NEED BE.
%\crdata{0-12345-67-8/90/01}  % Allows default copyright data (0-89791-88-6/97/05) to be over-ridden - IF NEED BE.
% --- End of Author Metadata ---

\title{Business Case In Software Repositories}

% You need the command \numberofauthors to handle the 'placement
% and alignment' of the authors beneath the title.
%
% For aesthetic reasons, we recommend 'three authors at a time'
% i.e. three 'name/affiliation blocks' be placed beneath the title.
%
% NOTE: You are NOT restricted in how many 'rows' of
% "name/affiliations" may appear. We just ask that you restrict
% the number of 'columns' to three.
%
% Because of the available 'opening page real-estate'
% we ask you to refrain from putting more than six authors
% (two rows with three columns) beneath the article title.
% More than six makes the first-page appear very cluttered indeed.
%
% Use the \alignauthor commands to handle the names
% and affiliations for an 'aesthetic maximum' of six authors.
% Add names, affiliations, addresses for
% the seventh etc. author(s) as the argument for the
% \additionalauthors command.
% These 'additional authors' will be output/set for you
% without further effort on your part as the last section in
% the body of your article BEFORE References or any Appendices.

\numberofauthors{2} %  in this sample file, there are a *total*
% of EIGHT authors. SIX appear on the 'first-page' (for formatting
% reasons) and the remaining two appear in the \additionalauthors section.
%
\author{
\alignauthor
Ekrem Kocaguneli, Gregory Gay\\Tim Menzies\\
       \affaddr{Lane Department of Computer Science and Electrical Engineering}\\
       \affaddr{West Virginia University}\\
       \affaddr{Morgantown, WV 26505, USA}\\
       \email{ekocagun@mix.wvu.edu, greg@greggay.com, tim@menzies.us}
% 2nd. author
\alignauthor Ye Yang\\
       \affaddr{Institute of Software}\\
	   \affaddr{Chinese Academy of Sciences}\\       
       \affaddr{Beijing, People's Republic of China }\\
       \email{ye@itechs.iscas.ac.cn}
}

\maketitle
\begin{abstract}
~\\\textit{\textbf{Background}}:
Due to inherent problems regarding within-company data like excessive time required to collect enough data, it is tempting to use cross-company data.
However, reliability of effort estimates based on cross-company data is contraversial. 
\\\textit{\textbf{Aim}:} 
We aim to find under what circumstances individual organizations can make use of cross-company data and question the effect of relevancy filtering, which has previously been proven significantly beneficial in software defect prediction domain.  
\\\textit{\textbf{Method}:}
We experiment the effects of relevancy filtering on 3 highly elaborated effort estimation datasets and their subsets.
We exploit the essential assumptions of analogy-based effort estimation and use a relevancy filtering model on the basis of performance variance.
\\\textit{\textbf{Results}:}
We have analyzed \textit{3 datasets * 3 subsets = 9 treatments} subject to statistical tests and found out that in 7 out of 9 treatments cross-company data performed as well as within company data.
\\\textit{\textbf{Conclusion}:}
Essential analogy-based effort estimation techniques aided with relevancy filtering on the basis of performance variance can enable organizations utilize large repositories of cross-company effort datasets.
\end{abstract}

%A category including the fourth, optional field follows...
\category{D.2.9}{Software Engineering}{Effort Estimation}[Management, measurement]

\keywords{Effort Estimation, estimation models, cross-company data, within-company data, software engineering}

\section{Introduction}

Effort estimation is one of the most important activities that a software development firm will undertake. 
Therefore, it is essential that such estimation be accurate. 
Serious consequences await projects with inaccurate estimates. 
The project's cost could eclipse its initial budget, or worse, may have to be canceled before it has been completed. 
Consider the case of NASA's Check-out Launch Control System, which was cancelled when the initial estimate of \$200 million was overrun by an {\em additional} \$200M \cite{CLCS03}. 
This case is not unique, despite the significant effort put into designing more accurate estimation models, it has been reported that many predictions are wrong by a factor of four or more \cite{Boehm1981, kemerer87}. 

We know from advice of Barry Boehm\footnote{Personal communication} that it is \textit{always} better to collect historical local data. 
Generating accurate effort estimates requires the use of such detailed and accurate historical data. 
Such a repository of historical records must be large enough to generate smooth, non over-fitted models, but it must also be regularly edited to remove projects based on obsolete technologies and methodologies that no longer fit with the company's standards. 
Constructing such a database involves some inevitable amount of effort. 
Project metrics must be tracked and regularly updated, along with the calculated effort totals, sometimes over the course of months or years. 
Such a task is difficult for an established company, and almost impossible for a new firm. 

In such cases, it would be desirable to use effort estimation data from some other source, perhaps even from another company. 
Such databases exist; the PROMISE repository \cite{Boetticher2007}, for instance, offers twelve effort estimation data sets for public access. 
However, tempting as such data can be, cross-company data comes with its own set of issues \cite{Mendes2005, Kitchenham2007, Mendes2004}. 
Accurate effort estimation functions on a theory of {\em locality}. 
New projects that follow similar practices to historical projects should require a similar amount of effort. 
As Chen et. al. \cite{chen09} have shown, inconsistencies in data collection across multiple firms create company-specific biases in cross-company data sets. 
Such biases result in an unacceptable amount of variance in the effort calculations. 
This same problem exists in the field of defect prediction. 
Cross-company data sources are riddled with inconsistencies, which leads to inaccurate predictions. However, multiple authors \cite{Turhan2009, Zimmermann2009} have shown that it is acceptable to use cross-comapny data sources, which have been preprocessed by some sort of {\em relevancy filtering}, either automated or expert-guided. 
Such filtering removes instances that create noise in the estimation process, leaving a body of data that, in theory, follows the principle of locality. 

We want to know if such relevancy filtering could be applied in an effort estimation context. 
Is it feasible to {\em collect} data from a variety of firms, {\em filter} it for heavily biased measurements, and then {\em apply} it to testing data from an unrelated company. 
Our research is guided by the following research questions:
\bi
\item[RQ1] What is the effect of selecting particular cross-company projects on estimation performance?
\item[RQ2] Is there any evidence that cross-company data can yield accuracy values as high as within-company data?
\item[RQ3] Do the characteristics of a particular dataset have an inf uence on the within-company versus cross-company performance?
\ei

In this paper, we demonstrate a tree-based relevancy filter that prunes the instances responsible for a large amount of variance in prediction quality. 
Using three cross-company datasets, each split into three firms, we demonstrate that cross-company data processed by our relevancy filter leads to more accurate predictions in over 75\% of experimental cases. 

\section{Background}

\subsection{Experts vs. Models}

Software effort estimation methods can be grouped under 2 categories\cite{Shepperd2007}: Expert judgment and model-based techniques.

Expert judgment methods: They are one of the most widely used estimation methods\cite{Jorgensen2004}.
Application of expert judgment may be either explicit (by following a method like Delphi\cite{Boehm2000}) or implicit (informal discussions among experts).
One problem with expert-based estimates is that they may fall short of reconciling competing interests, in the sense that estimation of a senior expert may be prevalent to that of a junior expert in a company setting.
Another problem indicated by Jorgensen et. al. is the poor capability of humans in improving their expert judgment\cite{Jorgensen2009}. 

Model-based techniques: These are the methods generated by using algorithmic/parametric approaches or induced prediction systems.
Former approach is the adaptation of an expert proposed model to local data.
A well known example to such an approach is Boehm's COCOMO method\cite{Boehm1981}.
The latter approach is useful in case that local data does not conform to the specifications of the expert's algorithmic/parametric method.
A few examples to induced prediction systems are linear regression, neural nets, model trees and analogy \cite{Menzies2006, Kadoda2001}.
All these systems are built on their inherent assumptions.
In case data violates these assumptions, patches are applied such as taking the logarithm of exponential distributions before linear regression \cite{Boehm1981, Kitchenham2009}.
However, choosing the appropriate patch again requires qualified experts.

Different organizations may choose to use an expert judgment, a model-based approach or a combination of the two in different settings.
However, the goal of an estimation model is usually common: To attain high accuracies.
Accuracy in estimates on the other hand, does not only depend on the choice of the estimation model.
Another critical factor towards accurate estimates is the detailed and correct historical data that are controlled time to time for being up to date.
 
\subsection{Within-Company vs. Cross-Company}

When a model-based techniqe is chosen for estimation, historical data come from past projects.
Many organizations may prefer to collect and maintain information concerning their own past projects in a dataset, so called within-company dataset.
Some of the previous studies have also suggested using company specific past data for accurate estimates \cite{kemerer87, Kitchenham1984}.
However, collection of a company specific data comes with a cost.
As experts on the field, in various settings Kitchenham and Mendes have reported 3 likely problems to occur when an organization wishes to rely on company specific data \cite{Mendes2004, Mendes2005, Kitchenham2007}:

\bi
\item[1)] Required time to accumulate enough local data may be prohibitive.
\item[2)] When local dataset is large enough, the technologies employed by the company may have changed and old projects may have become obselete.
\item[3)] For collecting local data in a consistent manner, care is necessary.
\ei

On top of all the afore mentioned right reasons, based on our personal experience, we can say that in the result oriented environment of today's business life, too much time spent in data collection and therefore postponing tangible results are likely to decrease enthusiasm of managers for the estimation practices.

The afore mentioned problems regarding local data have motivated researchers to search for alternative solutions like the use of cross-company datasets (datasets containing data from several different companies) \cite{Mendes2004, Mendes2005}.   
Although software effort estimation is a relatively young research field, collective effort of many contributors have enabled us to have a considerable repository of software effort datasets.
For example in PROMISE data repository alone, currently there are 12 effort prediction datasets that are open to public access \cite{Boetticher2007}.
However, use of cross-company datasets is not a silver-bullet solution and comes with its own problems\cite{Mendes2004, Mendes2005, Kitchenham2007}:
\bi
\item[1)] It is even more difficult to ensure the consistency of data collection accross multiple organizations.
\item[2)] Process and practices, hence trends may differ accross companies.
\item[3)] In the absense of a proper sampling strategy, it is difficult to make sure that selected cross company projects form a random sample of a defined population.
\ei

After mentioning the pros and cons on the issue of within vs. cross-company datasets, we shall take a look at the reflections on practice.
In their extensive systematic review on the issue, Kitchenham and Mendes have found contraversial claims \cite{Kitchenham06asystematic, Kitchenham2007} concerning results as well as methodologies of studies adressing the use of cross-company data in software effort estimation.
Among 10 studies under consideration in their 2006 study, only 7 of them had independent evidence in their comparisons of accuracies between cross-company and within company prediction models.
Within valid 7 studies, 3 studies found that cross-company model performance was not significantly worse than within company results, whereas the remaining 4 found that within-company models significantly outperformed cross-company models\cite{Kitchenham06asystematic}.
Therefore, the problem of cross-company vs. within-company is inconclusive under the light of previous studies and reviews. 

\section{Motivation}

Cross-company data use for prediction purposes is not limited to software effort estimation.
The subject has also been addressed in software defect prediction domain.
Turhan et. al. have questioned the applicability of cross-company data in software defect prediction domain and have shown in their research that cross-company can be successfully used \cite{Turhan2009}.
They have reported that conditions under which cross-company data can be used \textit{"as is"} are relatively limited\cite{Turhan2009}.
However, cross-company data when combined with relevancy filtering through an NN-filter have yielded surprisingly high accuracy values\cite{Turhan2009}.
Furthermore, Zimmermann et. al. have addressed the same question in individual project level and have come up with similar conclusions \cite{Zimmermann2009}.
They have defined the cross-prediction models as a serious challenge and they have reported that using the cross-company data as is does not lead to accurate predictions \cite{Zimmermann2009}.
To increase accuracy values Zimmermann et. al. have also proposed an approach that selects projects to be used for cross-company predictors \cite{Zimmermann2009}, which can be regarded as an expert guided relevancy filtering.
Taking these findings in defect prediction domain as a hint, we questioned the applicability and performance of alternative data processing techniques in software effort estimation domain.

\section{Approach}

\subsection{ABE0}
\label{subsection:ABE0}

Analogy-based estimation (ABE) in simplest terms generates an estimate for a test project by adapting the effort values of similar past projects in the train set.
By analyzing the previous research of experts such as Shepperd et. al.\cite{Shepperd1996}, Mendes et. al.\cite{Mendes2003} and Li et. al.\cite{Li2009} on the field of ABE, we can come up with a baseline for ABE:
\bi
\item Form a table whose rows are completed past projects, i.e. train set.
\item Columns of train set are composed of \textit{independent} variables (features defining projects) and a \textit{dependent} variable (recorded effort value). 
\item Decide on how many similar projects (\textit{analogies}) to use from train set, i.e. \textit{k}-value.
\item For each test instance, select \textit{k} analogies out of train set.
\bi
\item While selecting analogies, use a similarity/distance measure.
\item Before calculating similarity, apply a scaling measure on independent features to equalize their influence on the similarity measure.
\item Use a feature weighting scheme to reduce influence of less informative features.
\ei
\item Adapt effort values of \textit{k} nearest analogies to come up with an effort estimate.
\ei
We can refer to this baseline framework as ABE0.
ABE0 uses Euclidean distance as the similarity/distance measure, whose formula is given in Equation \ref{equ:euclid}.

\begin{equation}\label{equ:euclid}\footnotesize
Distance = \sqrt{\sum_{i=1}^{n}{w_i(x_{i} - y_{i})^{2}}}
\end{equation}

In Equation \ref{equ:euclid}, $w_i$ corresponds to feature weights applied to independent features.
ABE0 uses a uniform weighting, i.e. $w_i=1$.
Furthermore, the adaptation strategy for ultimate estimate is not necessarily a complex process.
ABE0 returns the median of the effort values of \textit{k} nearest analogies. 

\subsection{Relevancy Filtering}
\label{subsection:relevancyFiltering}

The existence of relevancy filtering has been shown to increase the accuracies in cross-company models in software defect prediction domain \cite{Turhan2009, Zimmermann2009}. 
Furthermore, using cross-company data in an \textit{as is} fashion has been shown to have a negative effect on accuracy values \cite{Zimmermann2009}.
Although different researchers have used different relevancy filters (Turhan et. al. use NN-filter, whereas filtering of Zimmermann et. al is more close to guided expert judgment), it is obvious that using only relevant data is helping to increase accuracies in cross-company models in defect prediction domain.

We have described an analogy-based estimation baseline (ABE0) in Section \ref{subsection:ABE0}.
As our relevancy filtering, we will use a form of ABE0 in which selection of analogies will be based on performance variance.

In our relevancy filtering approach, the training projects are used to generate a binary tree.
The leaves of the binary tree (level zero of the tree) are formed by individual training projects, which are then greedily grouped in tuples to form a binary.
The first binary tree (BT1) is then traversed from root to level one (one higher level than leaves).
Variance of effort values in each sub-tree (performance variance) is then recorded and scaled to 0-1 interval.
Then a random relevancy filtering is applied on scaled variances as follows:
\bi
\item Generate a random variable from a normal distribution.
\item Prune nodes whose scaled performance variance is above the generated random variable.
\ei

The remaining training instances are used to build a second binary tree (BT2) and are then subject to a second relevancy filtering.
BT2 is generated in the same fashion as BT1 and traversed from root to level 1 of the tree.
But this time while traversing from the root down the tree, instead of storing the variances of sub-trees, we use the variance as a decision criterion: If the variance of the current-tree is larger than its sub-trees, then move down the subtree; otherwise, stop moving and select the instances in the current-tree as the relevant instances and adapt them for estimation.
Since the described model is a version of ABE0, adaptation used for the selected/relevant instances of BT2 is the same as ABE0, i.e. taking median.
A simple visualization of this relevancy filtering approach is given in in Figure \ref{fig:filtering}.

\begin{figure}
\begin{center}
\includegraphics[width=0.35\textwidth]{filtering.png}
\end{center}
\caption{A representation of relevancy filtering. RR region contains instances selected by \textbf{R}andom \textbf{R}elevancy filtering in BT1 tree and SR region has the instances of \textbf{S}elective \textbf{R}elevancy filtering for a particular test instance in BT2.}\label{fig:filtering}
\end{figure}

The afore mentioned relevancy filtering is very similar to NN-filtering except the fact that for each test instance the number of analogies to be selected (\textit{k}-value) is determined online depending on the characteristics of the current train set.
Furthermore, unlike models proposed in previous cross-company research, relevancy filtering on the basis of performance variance makes no pre-assumptions and lets the test instance adapt to characteristics of the train set to prune irrelevancies.



\section{Methodology}

Some of the previous studies were criticized due to their lack of proper statistical analysis or due to limited datasets they have used \cite{Kitchenham2007}.
Therefore, while conducting our research we paid particular attention to eliminate previouly reported issues. 
The details regarding dataset selection criteria as well as statistical analysis method we have adopted in this research will be provided in Seciton \ref{subsection:Data} and Section \ref{subsection:Experiments} respectively.

Reproducibility of results is another critical issue in software engineering \cite{Juristo2009b} and another criticism to previous work is lacking the information to be reproducable \cite{Kitchenham06asystematic, Kitchenham2007}.
Since we want our results to be reproduced in similar or different settings, we chose publicly available datasets and gave as much details as possible regarding our experimental settings.

\subsection{Data}
\label{subsection:Data}

In our research, we have used subsets of 3 commonly used datasets in software effort estimation: Nasa93, original Cocomo81 \cite{Boehm1981} and Desharnais datasets \cite{Desharnais1989}.

We will denote the subsets of Nasa93 as Nasa93c1, Nasa93c2 and Nasa93c5.
Nasa93c1, Nasa93c2 and Nasa93c5 come from different development center around the USA (from development center 1, 2 and 5 respectively).
In a similar fashion, subsets of Cocomo81 will be denoted by Coc81o, Coc81e and Coc81s:
\bi
\item
Coc81o contains ``organic projects'' that come from
small teams with good experience working with less than rigid requirements.
\item Coc81e contains ``embedded projects'' that
are developed within tight constraints (hardware, software, operational etc.).
\item The ``semidetached projects'' denoted by Coc81s are at an intermediate stage between the organic and embedded modes.
\ei

Lastly Desharnais dataset will have 3 different subsets depending on 3 different programming languages used for developing projects in each subset. We will denote subsets of Desharnais as DesL1, DesL2 and DesL3 which were developed in languages 1, 2 and 3 respectively.
The subsets of each dataset are chosen to form a self consistent and reliable data groups.
Since each of these subsets have certain common criteria such as development center, development mode and development language, each subset will be treated as a within-company dataset.
The details such as feature and instance sizes as well as content of datasets are given in Figure \ref{fig:data}.
All of the datasets used in this research are available in PROMISE data repository \cite{Boetticher2007}.

\begin{figure*}
\centering
{\scriptsize  
\begin{tabular}{l|rrl|l}  
 \textbf{} & \textbf{} & \textbf{} & \textbf{}&\multicolumn{1}{c}{\textbf{Historical Effort Data}}\\
 \textbf{Dataset} & \textbf{Features} & \textbf{Projects} & \textbf{Content}&\textbf{Units}\\ \hline
  Cocomo81 &17&63& NASA projects& months\\
  \hspace{6 mm}Coc81o &17&24& Cocomo81 organic projects&months\\ 
  \hspace{6 mm}Coc81e &17&28& Cocomo81 embedded projects&months\\ 
  \hspace{6 mm}Coc81s &17&11& Cocomo81 semidetached projects&months\\   
  Nasa93 &17&93& NASA projects&months\\ 
  \hspace{6 mm}Nasa93c1 &17&12& Nasa93 projects from center 1&months\\ 
  \hspace{6 mm}Nasa93c2 &17&37& Nasa93 projects from center 2&months\\ 
  \hspace{6 mm}Nasa93c5 &17&39& Nasa93 projects from center 5&months\\ 
  Desharnais &12&81& Canadian software projects&hours\\
  \hspace{6 mm}DesL1 &11&46& Desharnais projects developed with language 1&hours\\ 
  \hspace{6 mm}DesL2 &11&25& Desharnais projects developed with language 2&hours\\
  \hspace{6 mm}DesL3 &11&10& Desharnais projects developed with language 3&hours\\  \cline{3-3}
  \multicolumn{2}{c}{~}  & Total: 469& \multicolumn{7}{c}{~} 
\end{tabular}}
\caption{The 469 projects used in this study come from \textit{3 datasets + 9 subsets = 12} data sets.
Indentation in column one denotes that indented dataset is a subset of another dataset.
}\label{fig:data}
\end{figure*}

Kitchenham and Mendes attributes a particular importance to dataset size in their review \cite{Kitchenham2007} and say that larger within company datasets lead to more reliable comparisons between within-company and cross-company models.
To evaluate the goodness of within-company datasets, they propose a quality scoring of 4 values: Poor quality (less than 10 projects), fair quality (between 10 to 20 projects), good quality (between 20 to 40 projects) and excellent quality (more than 40 projects) \cite{Kitchenham2007}.
According to quality criteria indicated by Kitchenham et. al. \cite{Kitchenham2007}, 9 within-company datasets used in this study are grouped as follows: 1 excellent quality, 5 good quality and 3 fair quality datasets.

\subsection{Experiments}
\label{subsection:Experiments}

For each of the 3 main datasets (Nasa93, Cocomo81 and Desharnais) in our research we conduct within-company and cross-company experiments.
The main category of division for a main datasets into its subsets is for the subsets to have a self consistent structure according to a dataset specific characteristic.
Therefore, each subset can be considered similar to a within-company dataset that contains projects sharing a particular characteristic in common.

\begin{figure*}[ht!]
\begin{center}
\includegraphics[width=0.8\textwidth]{ccexperiments.png}
\end{center}
\caption{In cross-company experiments for subsets $X_1$, $X_2$ and $X_3$ we have 3 treatments. At each treatment, one subset is selected as test set and the remaining two become the cross-company data, i.e. train set. After filtering, median of the remaining instances from the train set become our estimate.}\label{fig:ccexperiments}
\end{figure*}

To understand the within-company and cross-company data formation, assume that a dataset $X$ with its 3 subsets $X_1$, $X_2$ and $X_3$ is under consideration.
For within-company experiments, relevancy filtering described in Section \ref{subsection:relevancyFiltering} is applied on each one of $X_1$, $X_2$ and $X_3$ separely and the median of filtered project instances in the train set is stored as the effort estimate for the test instance.
For the separation of training and test sets, leave-one-out method is used.
Leave-one-out basically selects one instance out of a dataset of $n$ instances as the test instance and uses the remaining $n-1$ instances as the train set.

As for the cross-company experiments, one of $X_1$, $X_2$ and $X_3$ is chosen as the test set and the combination of the remaining two forms the cross-company dataset.
This time relevancy filtering is applied on the cross-company dataset, then estimations for projects in the test set are stored and this completes a single run.
An illustration showing how the cross-company experiments are performed is given in Figure 

The above described within-company and cross-company runs are repeated 20 times to remove the bias that would otherwise be brought by a particular test and train set combination. 

\subsection{Performance Criteria}

To be able to compare the performance of within-company and cross-company datasets subject to relevancy filtering, we used two measures: magnitude of relative error (MRE) and win-tie-loss values.
The reason behind using MRE is fact that it is the most commonly used performance criterion for software effort estimation \cite{Molokken-Ostvold2004}.
Furthermore, as we see from Formula \ref{EquationMRE}, MRE gives a per instance based estimation performance evaluation.

\begin{equation} 
MRE = \frac{|actual_{i} - predicted_{i}|}{actual_{i}}
\label{EquationMRE} 
\end{equation}

\begin{figure}
\renewcommand{\baselinestretch}{1}
\begin{algorithmic} \footnotesize
\STATE \textit{$win_{i}$} = 0, \textit{$tie_{i}$} = 0, \textit{$loss_{i}$} = 0
\STATE \textit{$win_{j}$} = 0, \textit{$tie_{j}$} = 0, \textit{$loss_{j}$} = 0
\IF{MANN-WHITNEY(\textit{$MRE's_{i}$}, \textit{$MRE's_{i}$}) says they are the same}
	\STATE \textit{$tie_{i}$} = \textit{$tie_{i}$} + 1;
\ELSE
	\IF{median(\textit{$MRE's_{i}$}) $<$ median(\textit{$MRE's_{j}$})}
		\STATE \textit{$win_{i}$} = \textit{$win_{i}$} + 1
		\STATE \textit{$loss_{j}$} = \textit{$loss_{j}$} + 1
	\ELSE
		\STATE \textit{$win_{j}$} = \textit{$win_{j}$} + 1
		\STATE \textit{$loss_{i}$} = \textit{$loss_{i}$} + 1
	\ENDIF
\ENDIF
\end{algorithmic}
\caption{Pseudocode for Win-Tie-Loss Calculation Between Method \textit{i} and \textit{j}}
\label{FigureWinTieLossPseudocode}
\end{figure}

However, MRE is subject to many pitfalls.
When MRE is used as a stand-alone evaluation criterion and when not combined with appropriate statistical tests, it may lead to biased or even false conclusions.
Not to fall into MRE related pitfalls, we use another performance criterion called win-tie-loss calculation.
Win-tie-loss calculation states the fact that comparison between two methods $i$ and $j$ makes sense only if they are statistically significant.
If they are statistically the same, they are in fact observations coming from the same distribution, therefore they tie and their $tie$ values ($tie_i$ and $tie_j$) are incremented.
On the other hand, if there is a statistical difference between two methods, then the method with a lower median MRE score, say $i$,  wins and the one with the lower MRE , say $j$, loses; hence, the related values $win_i$ and $loss_j$ are incremented by one.
Since we make 20 runs for each treatment, win-tie-loss calculation provides us a good perspective on the success of each method for different datasets.
The pseudocode of win-tie-loss is given in Figure \ref{FigureWinTieLossPseudocode}.
For the comparison of methods in win-tie-loss calculation a non-parametric statistical test Mann-Whitney is used at a significance level of 95\%.


\section{Results}

In our research we analyzed \textit{3 datasets * 3 subsets = 9 treatments}.
In each treatment, we applied relevancy filtering on training data on the basis of performance variance and evaluated cross-company as well as within-company performances of a particular dataset subject to statistical tests.

\subsection{Preliminary Results}

Depending on the earlier results as well as related literature reviews \cite{Turhan2009, Zimmermann2009, Mendes2004, Mendes2005, Kitchenham06asystematic, Kitchenham2007}, we were quite aware of the fact that making use of cross-company or within-company data for software effort estimation is a contraversial issue.
Therefore, before going into details of the performance measures regarding within-company and cross-company datasets, we wanted to observe the behaviour of cross-company projects from different datasets when subject to relevancy filtering.

In Figure \ref{fig:srsizeNasa93} we see the effect of relevancy filtering for Nasa93 dataset: Instances of cross-company dataset that come from different centers are not favored over one another.
On the contrary, some instances were selected from each center forming the cross-company dataset.
Under second column in Figure \ref{fig:srsizeNasa93}, we see how many instances were selected on average from each center in each cross-company dataset.
Although we see that relevancy filtering chooses certain projects from all centers of cross-company datasets, we are unable to observe a definite pattern.

\begin{figure}[th!]
\renewcommand{\baselinestretch}{.8}
\centering
\footnotesize
  \begin{tabular}{ l | c  c  c }   
  \textbf{Cross-Companies} & \multicolumn{3}{c}{\textbf{Selected}} \\\hline	
    &		&		&		\\
	&		Nasa93c1	&	Nasa93c2	&	Nasa93c5	\\
Nasa93c2 and Nasa93c5	&		0	&	2.35	&	1.17	\\
Nasa93c1 and Nasa93c5	&		1.41	&	0	&	1.79	\\
Nasa93c1 and Nasa93c2	&		2.36	&	1.08	&	0	\\
  &		&		&		\\
  	\end{tabular}
\caption{After relevancy filtering on cross-company datasets, the distribution of \textit{selected} instances do not come from a particular center of Nasa93.}
\label{fig:srsizeNasa93}
\end{figure}

Note that in Figure \ref{fig:srsizeNasa93} each row has one center whose selected instance number is zero.
This is due to the fact that one particular center is kept as the test set; hence no instances from that center can be selected for that row.

Figure \ref{fig:srsizeCoc81} shows the average number of projects selected from each subset in cross-company datasets of Cocomo81.
The same effect of relevancy filtering that we have seen on Nasa93 can also be observed on Cocomo81.
As we can see in Figure \ref{fig:srsizeCoc81}, relevancy filtering selects instances from all subsets forming a particular cross-company dataset.
For the last treatment (Coc81o and Coc81e) in Figure \ref{fig:srsizeCoc81}, we see that the effect of relevancy filtering for selecting instances from each subset is quite limited, i.e. the average number of instances chosen from Coc81o is 2.75 whereas the same number for Coc81e is 0.25.


\begin{figure}[th!]
\renewcommand{\baselinestretch}{.8}
\centering
\footnotesize
  \begin{tabular}{ l | c  c  c }   
  \textbf{Cross-Companies} & \multicolumn{3}{c}{\textbf{Selected}} \\\hline	
    &		&		&		\\
	&		Coc81o	&	Coc81e	&	Coc81s	\\
Coc81e and Coc81s	&			0	&	1.98	&	1.29	\\
Coc81o and Coc81s	&			3.57	&	0	&	1.36	\\
Coc81o and Coc81e	&			2.75	&	0.25	&	0	\\
  &		&		&		\\
  	\end{tabular}
\caption{Effect of relevancy filtering on Cocomo81 cross-companies is similar to those of Nasa93. Instances are selected from all subsets.}
\label{fig:srsizeCoc81}
\end{figure}

Figure \ref{fig:srsizeDesharnais} also describes the same effect we have seen previously in Figure \ref{fig:srsizeNasa93} and Figure \ref{fig:srsizeCoc81}: Selection of instances are not limited to a single subset in cross-company datasets.
However, if we look at row two (DesL1 and DesL3) of Figure \ref{fig:srsizeDesharnais}, we see that this effect is extremely limited: An average of 2.05 instances from DesL1 and only 0.1 instances from DesL2.
We have seen such a situation in Cocomo81 cross-company datasets as well.
Therefore, we may suggest that there is a hidden influence of certain dimensions/features on the selection of instances from cross-company datasets.
However, we will leave the task of discovering these features and defining their influences as the subject of a new study, since that work requires a research in its own right.

\begin{figure}[th!]
\renewcommand{\baselinestretch}{.8}
\centering
\footnotesize
  \begin{tabular}{ l | c  c  c }   
  \textbf{Cross-Companies} & \multicolumn{3}{c}{\textbf{Selected}} \\\hline	
    &		&		&		\\
	&		DesL1	&	DesL2	&	DesL3	\\
DesL2 and DesL3	&	0	&	2.27	&	0.65 \\
DesL1 and DesL3	&	2.05	&	0	&	0.1 \\
DesL1 and DesL2	&	3.17	&	1.59	&	0 \\

  &		&		&		\\
  	\end{tabular}
\caption{Similar trend of relevancy filtering continues for Desharnais dataset as well. Selected instances come from all datasets of cross-company datasets.}
\label{fig:srsizeDesharnais}
\end{figure}

As a result of our preliminary results, we have seen that relevancy filtering on the basis of performance variance does not favor any certain subset of cross-company datasets. 
On the contrary, the instances selected as relevant, i.e. instances in SR tree in Figure \ref{fig:filtering} come from all the companies that compose the cross-company dataset.
The the summary of our preliminary results is given in Figure \ref{fig:srsize}.
We can observe from Figure \ref{fig:srsize} that for each cross-company dataset, there are instances from all subsets being selected by relevancy filtering.
As it seems, although we have based our definition of within-company data on single dimensions such as geographical proximity of development teams (centers 1, 2 and 5 of Nasa93), mode of development (organic, embedded and semi-detached modes of Cocomo81) or particular languages used to develop projects (languages 1, 2 and 3 of Desharnais), there are yet other factors that are also influential on the selection of projects as relevant to a test instance. 

\begin{figure*}[th!]
\renewcommand{\baselinestretch}{.8}
\centering
\footnotesize
  \begin{tabular}{ l | l | c  c  c }   
  \textbf{Cross-Company Dataset} & \textbf{Test Set}  & \multicolumn{3}{c}{\textbf{Instance Size in SR Tree}} \\\hline	
    &		&		&		&		\\
	&		&	From Nasa93c1	&	From Nasa93c1	&	From Nasa93c1	\\
Nasa93c2 and Nasa93c5	&	Nasa93c1	&	0	&	2.35	&	1.17	\\
Nasa93c1 and Nasa93c5	&	Nasa93c2	&	1.41	&	0	&	1.79	\\
Nasa93c1 and Nasa93c2	&	Nasa93c5	&	2.36	&	1.08	&	0	\\
  &		&		&		&		\\\hline
  &		&		&		&		\\									
	&		&	From Coc81o	&	From Coc81e	&	From Coc81s	\\
Coc81e and Coc81s	&	Coc81o	&	0	&	1.98	&	1.29	\\
Coc81o and Coc81s	&	Coc81e	&	3.57	&	0	&	1.36	\\
Coc81o and Coc81e	&	Coc81s	&	2.75	&	0.25	&	0	\\
  &		&		&		&		\\\hline
		  &		&		&		&		\\							
	&		&	From DesL1	&	From DesL2	&	From DesL3	\\
DesL2 and DesL3	&	DesL1	&	0	&	2.27	&	0.65	\\
DesL1 and DesL3	&	DesL2	&	2.05	&	0	&	0.1	\\
DesL1 and DesL2	&	DesL3	&	3.17	&	1.59	&	0	\\
	\end{tabular}
\caption{The instance sizes in SR tree are mean of remaining instances after filtering in 20 runs. Cross-company datasets are combination of two within-company datasets tested on another within-company dataset. We see that relevancy filtering selects instances from both centers forming a cross-company dataset.}
\label{fig:srsize}
\end{figure*}

\subsection{Performance Results}

The preliminary results motivated us to go further with the experiments and question how the effect observed in preliminary results reflected on the performances of cross-company datasets.

In Figure \ref{fig:wltNasa93} we see the win-tie-loss values for subsets of Nasa93 subject to 3 treatments.
Figure \ref{fig:wltNasa93} shows us that in all 3 treatments the $tie$ values are quite high.
The $tie$ values between within-company and cross-company datasets of Nasa93 are 19, 17 and 15, which means that at least 75\% of the times there is no statistical difference between cross-company and within company results.
Even for the treatment with the lowest $tie$ value of 15, within-company has a $win$ value of 3 and cross-company has a $win$ value of 2; i.e. for Nasa93, performance of cross-company is not distinguishable from the performance of within-company.

\begin{figure}[th!]
\renewcommand{\baselinestretch}{.8}
\centering
\footnotesize
  \begin{tabular}{l | l | c  c  c }   
  \textbf{Dataset} & \textbf{Method} & \textbf{Win} & \textbf{Tie} & \textbf{Loss} \\ \hline
  & & & \\			
Nasa93c1	&	WC-Data	& 1	&	    19	&	     0 \\
Nasa93c2 and Nasa93c5	&	CC-Data	&	     0	&	    19	&	     1	\\
 & & & \\	\hline
   & & & \\	
Nasa93c2	&	WC-Data	&	3	&	17	&	3 \\
Nasa93c1 and Nasa93c5	&	CC-Data	&	0	&	17	&	0	\\
 & & & \\	\hline
   & & & \\	
Nasa93c5	&	WC-Data	&	3	&	15	&	2	\\
Nasa93c1 and Nasa93c2	&	CC-Data	&	2	&	15	&	3	\\
	\end{tabular}
\caption{MRE win-tie-loss values for Nasa93 from 20 {\em randomized assessments}. In all treatments $tie$ values are quite high. For Nasa93, the performance of cross-company data (CC-Data) is mostly same as within-company data (WC-Data).}
\label{fig:wltNasa93}
\end{figure}

Figure \ref{fig:wltCocomo81} shows the win-tie-loss values for subsets of Cocomo81.
In 2 out of 3 treatments the $tie$ values are 19, which tells us that for these treatments within-company and cross-company datasets are almost identical.
On the other hand, for the last treatment within-company dataset of Coc81s statistically performs better than the cross-company dataset for 13 times out of 20 runs.
If we take a look at Figure \ref{fig:srsizeCoc81} in our preliminary results, we see that in comparison to other within-company datasets of Cocomo81, Coc81s was more selective towards a certain domain (selection of 2.75 from Coc81o and only 0.25 from Coc81e) under relevancy filtering.\footnote{There is a similar connection for Desharnais language 1 wc dataset..but I cannot see how we can describe it. Or we can just ignore this connection.. Any help?}

\begin{figure}[th!]
\renewcommand{\baselinestretch}{.8}
\centering
\footnotesize
  \begin{tabular}{l | l | c  c  c }   
  \textbf{Dataset} & \textbf{Method} & \textbf{Win} & \textbf{Tie} & \textbf{Loss} \\ \hline
  & & & \\			
Coc81o	&	WC-Data	&	0	&	20	&	0	\\
Coc81e and Coc81s	&	CC-Data	&	0	&	20	&	0	\\
 & & & \\	\hline
   & & & \\	
Coc81e	&	WC-Data		&	1	&	19	&	0	\\
Coc81o and Coc81s	&	CC-Data		&	0	&	19	&	1	\\
 & & & \\	\hline
   & & & \\	
Coc81s	&	WC-Data		&	    13	&	7	&	0	\\
Coc81o and Coc81e	&	CC-Data	& 0	&	7	&	13	\\
\end{tabular}
\caption{MRE win-tie-loss values for Cocomo81 from 20 {\em randomized assessments}. In 2 treatments CC-Data is same as WC-Data. However, in the case of Coc81s WC-Data outperforms CC-Data.}
\label{fig:wltCocomo81}
\end{figure}

The win-tie-loss values for subsets of Desharnais are given in Figure \ref{fig:wltDesharnais}.
Results for Desharnais subsets are similar to those of Cocomo81 subsets, in the sense that 2 out of 3 treatments have $tie$ values of 19, which suggests that performance of cross-company datasets are not statistically different from within-company datasets.
However, in 1 treatment within-company dataset outperforms cross-company.
In the case of DesL1, 16 out of 20 runs within-company dataset performs better than cross-company.

\begin{figure}[th!]
\renewcommand{\baselinestretch}{.8}
\centering
\footnotesize
  \begin{tabular}{l | l | c  c  c }   
  \textbf{Dataset} & \textbf{Method} & \textbf{Win} & \textbf{Tie} & \textbf{Loss} \\ \hline
  & & & \\			
DesL1	&	WC-Data	&	16	&	4	&	0	\\
DesL2 and DesL3	&	CC-Data	& 0	&	4	&	16	\\
 & & & \\	\hline
   & & & \\	
DesL2	&	WC-Data		&	1	&	19	&	0	\\
DesL1 and DesL3	&	CC-Data		&	0	&	19	&	1	\\
 & & & \\	\hline
   & & & \\	
DesL3	&	WC-Data		&	1	&	19	&	0	\\
DesL1 and DesL2	&	CC-Data	&	0	&	19	&	1	\\
		\end{tabular}
\caption{MRE win-tie-loss values for Desharnais from 20 {\em randomized assessments}. In the case of DesL1 WC-Data is much better than CC-Data. For other treatments, WC and CC-Data are statistically the same.}
\label{fig:wltDesharnais}
\end{figure}

The summary of our results in terms of win-tie-loss values are given in Figure \ref{fig:wlt}.
Since we compare cross-company and within-company performance in 20 randomized assessments for each treatment, the sum of $win$, $tie$ and $loss$ values at each row of Figure \ref{fig:wlt} is 20.
When we look at the win-tie-loss values, the first noticable fact is the high $tie$ values between cross-company and within-company models in each treatment.
In 7 out of 9 treatments which is more than 75\% of all the treatments, the $tie$ values are equal to or more than 15. 
Another interpretation of high $tie$ values is that in more than 75\% of the cases, there is not a statistical difference between cross-company and within-company datasets and cross-company datasets have performed as well as within-company datasets.
Only in two treatments, DesL1 and Coc81s, the within-company datasets were significantly better than cross-company datasets.
Possible explanation for this scenario may be hidden in the dataset size or in the data quality of within-company datasets, but of course with the available information at hand, it is difficult to suggest a conclusive reason for that situation.

\subsection{Answers to Research Questions}
In the light of these results, we can now answer the research questions we have proposed to guide this research.

\textbf{RQ1) What is the effect of selecting particular cross-company projects in estimation performance?}

We have seen that in more than 75\% of the treatments cross-company datasets have performed as well as within-company datasets.
Therefore, we can say that selecting particular cross-company projects on the basis of relevancy filtering subject to analogy-based estimation has a positive effect on the estimation performance.

\textbf{RQ2) Is there evidence that cross-company data can yield accuracy values as high as within-company data?}

In our experiments for 7 out of 9 treatments cross-company data have performed as well as within-company data. 
Therefore, we can suggest the existence of a statistical evidence that cross-company data can also yield high accuracy values.
However, for cross-company data to attain high accuracy values, we should avoid using it in an \textit{``as is"} manner.
In our case, the high accuracy values are due to relevancy filtering on the basis of performance variance, which filters out only the relevant instances that have low variance values.

\textbf{RQ3) Do characteristics of particular datasets have an influence on within-company versus cross-company performance?}

In two treatments, we have observed that within-company datasets significanlty outperformed cross-company datasets.
Therefore, we cannot say that characteristics of particular datasets have no influence on the within-company effort estimation performance.
Although being a minority, some datasets yield significantly higher estimation performance with within-company data.
However, the reason behind this fact is not obvious.
In our case, two treatments that yielded higher within-company performance were DesL1 and Coc81s datasets with 46 and 11 projects respectively. 
When considering the dataset sizes, it is difficult to observe a pattern between dataset size and within-company performance.
Another likely reason could be the data quality of the datasets.
However, with the available information regarding datasets, it is not possible to comment on the existence of a pattern with data quality too.


\begin{figure*}[th!]
\renewcommand{\baselinestretch}{.8}
\centering
\footnotesize
  \begin{tabular}{l | l  l  l | c  c  c }   
  \textbf{Data set} &\textbf{Train Set} & \textbf{Test Set} & \textbf{Method} & \textbf{Win} & \textbf{Tie} & \textbf{Loss} \\ \hline			
Nasa93	&	Nasa93c1	&	Leave-one-out test instance	&	Within-Company	& 1	&	    19	&	     0 \\
	&	Nasa93c2 and Nasa93c5	&	Nasa93c1	&	Cross-Company	&	     0	&	    19	&	     1	\\
	&	Nasa93c2	&	Leave-one-out test instance	&	Within-Company	&	3	&	17	&	3 \\
	&	Nasa93c1 and Nasa93c5	&	Nasa93c2	&	Cross-Company	&	0	&	17	&	0	\\
	&	Nasa93c5	&	Leave-one-out test instance	&	Within-Company	&	3	&	15	&	2	\\
    &	Nasa93c1 and Nasa93c2	&	Nasa93c5	&	Cross-Company	&	2	&	15	&	3	\\	\hline
Cocomo81	&	Coc81o	&	Leave-one-out test instance	&	Within-Company	&	0	&	20	&	0	\\	
	&	Coc81e and Coc81s	&	Coc81o	&	Cross-Company	&	0	&	20	&	0	\\
	&	Coc81e	&	Leave-one-out test instance	&	Within-Company	&	1	&	19	&	0	\\
	&	Coc81o and Coc81s	&	Coc81e	&	Cross-Company	&	0	&	19	&	1	\\
	&	Coc81s	&	Leave-one-out test instance	&	Within-Company	&	    13	&	7	&	0	\\
	&	Coc81o and Coc81e	&	Coc81s	&	Cross-Company	&	     0	&	7	&	13	\\	\hline
Desharnais		&	DesL1	&	Leave-one-out test instance	&	Within-Company	&	16	&	4	&	0	\\
	&	DesL2 and DesL3	&	DesL1	&	Cross-Company	&	     0	&	4	&	16	\\
	&	DesL2	&	Leave-one-out test instance	&	Within-Company	&	1	&	19	&	0	\\	
	&	DesL1 and DesL3	&	DesL2	&	Cross-Company	&	0	&	19	&	1	\\
	&	DesL3	&	Leave-one-out test instance	&	Within-Company	&	1	&	19	&	0	\\
	&	DesL1 and DesL2	&	DesL3	&	Cross-Company	&	0	&	19	&	1	\\
	\end{tabular}
\caption{MRE win-loss-tie results from 20 {\em randomized assessments}.
Comparisons are between within-company and cross-company performance of each subset of every dataset.}\label{fig:wlt}
\end{figure*}

\section{Threats to Validity}
We will address the threats to validity to our results under 2 categories: Internal validity and external validity.

Internal validity asks up to what degree the cause-effect relationship between dependent and independent variables hold \cite{Robson2002}.
The ideal case for satisfying internal validity would be to learn a theory from a past situation and then to apply the theory to a new setting.
However, in general software effort studies make use of commonly exploited past datasets such as the ones used in this research.
That is an internal validity issue that threats all effort studies using only past data.
On the other hand, we can still mitigate this problem by simulating the behaviour of a learned theory in new settings.
In our study we make use of leave-one-out method for all treatments to address internal validity issues. Leave-one-out enables us to separate the train and test sets completely in each experiment.

External validity is the generalizability of results.
A likely threat to external validity of our research is that we use subsets of 3 different datasets as our within-company data.
This approach may seem non-traditional when compared to previous cross-company studies.
However, each dataset we used in our study is composed of projects coming from different vendors with various different properties.
Therefore, to split datasets into subsets we chose certain dimensions that would both validate the subsets as cross-companies and also that would produce good quality datasets according to definition of Kitchenham et. al.\cite{Kitchenham2007}. 
For Nasa93 this dimension was pysical proximity (development centers), for Cocomo81 it was the organization of teams (development mode) and for Desharnais it was the language used to develop programs.

\section{Conclusion}
In our research we questioned the use of cross-company data in software effort estimation domain.
Our motivation was that instead of using cross-company data in an \textit{"as is"} manner, subjecting it to a filtering process could improve the estimation accuracy.
To observe the effect of relevancy filtering on cross-company datasets, we used performance variance on the binary trees.
Furthermore, as our estimation model we used a simple variant of ABE0.

We observed that our motivation had statistical validity.
If not in all of our treatments, in more than 75\% of them we have observed that cross-company data performance was as good as that of within-company data.
Therefore, it is possible to claim that the positive effect of relevancy filtering that was previously reported in defect prediction domain \cite{Turhan2009, Zimmermann2009} can also be found in software effort estimation domain.
 
We have also observed in 2 treatments that within-company data outperforms cross-company data.
However, since we cannot see a certain pattern in these cases, our account for them are closer to personal opinions than conclusive remarks.

As a result of our research, our suggestions to organizations willing to undertake a software effort estimation practice would be to benefit from the existence of cross-company datasets.
However, while using cross-company data, organizations should avoid using the whole dataset without considering relevancy of the particular projects.
Instead of using the whole cross-company data, organizations should consider using a relevancy filtering method.

In this research we showed that cross-company methods can attain estimation accuracies as high as those of within-company.
Of course the possibility of having a better within-company dataset always exists and it is always better to collect local data.
For instance, 2 out of 9 treatments in our research favored within-company data. 
However, at early stages of an effort estimation project local data may not be readily available.
Therefore, a good strategy to follow would be to start benefiting from available cross-company repositories until a point where a reliable and statistically better within-company dataset is formed.

\section{Future Work}

Going further on this research, we want to learn how relevant instances from different subsets in cross-company datasets are filtered out.
In other words, we would like to question which dimensions are most influential to the definition of within and cross company datasets.
Being able to define these dimensions could ease the selection of the most appropriate projects from different cross-company datasets for an institution that is willing to use cross-company datasets in an effort estimation practice.

\section{Acknowledgements}
This work was conducted under contract with the Institute of Software, Chinese Academy of Sciences.

\bibliographystyle{abbrv}
\bibliography{myref}  % sigproc.bib is the name of the Bibliography in this case

\end{document}
