\section{Datasets and Project Descriptions}

\begin{figure}[!h]
\centering
\begin{tabular}{l|rrl|l}  
 \textbf{Dataset} & \textbf{Cols} & \textbf{Rows} & \textbf{Notes}&\textbf{Measures }\\\hline
Kemerer	& 7 & 15 & Large business applications	& effort \\ 
Telecom	& 3 & 18 & U.K. telecom enhancements	& effort\\ 
Finnish	& 8 & 38 & Finnish IS projects		& effort \\
Miyazaki& 8 & 48 & Japanese COBOL projects	& effort\\ 
COC81ii	&26 & 63 & NASA projects 		& effort, time, defects\\
NASA93ii	&26 & 93 & NASA projects 		& effort, time, defects \\
China	&17 &499 & Chinese software projects	& effort \\\cline{3-3}
          \multicolumn{2}{c}{~}  & Total: 774\\
\end{tabular}
\caption{Seven data sets from \protect\url{promisedata.org/?cat=14}: {\em effort} is total staff person-months; {\em  time} is calendar time (start to stop);
{\em defects} represents the number of delivered defects.
}\label{fig:data}
\end{figure}

Because {\W}2 makes no underlying model assumptions, we aren't limited to the COCOMO II ontology for our performance evaluations. However, in order to compare {\W}2's case-based methodology with SEESAW's model-based methodology, these comparisons must be made within the same ontology. For the following experiments, any comparison with SEESAW will use the NASA93 and COC81 datasets.

Both NASA93 and COC81 initially contained only effort data in the original COCOMO format. However, using Boehm's COQUALMO defect prediction we have converted this data into COCOMO II format, which contains estimated total defect numbers as well as estimated completion time in months. COCOMO II also breaks down the project attributes into both scale factors and effort multipliers, outlined earlier in ~\fig{cocattrs}.

The other datasets use for these experiments is detailed in ~\fig{data}. For non-COCOMO data, the reduction goal is the same: software effort in person-months. These datasets include the following:
\bi
\item
Enhancements to a U.K. telecommunications product;
\item
Projects collected by Miyazaki et al~\cite{miyazaki94};
\item
Finnish Information Systems projects;
\item
A large dataset of Chinese software projects;
\item
Large COBOL projects, collected by Kemerer~\cite{kemerer87}.
\ei
The format of this data is highly varied and includes number of basic logical transactions, query count and number of distinct business units serviced. 

Project descriptions are also needed to define the space of controllable options. For the non-COCOMO data sets, we did not have access to specific case studies like \fig{cases}. Hence, these results are based on {\em contexts} developed as follows:

\bi
\item One project contained all possible project values as $controllables$
\item The remaining two projects featured randomly chosen ranges with ranges half of the possible values.
\ei
Data and project descriptions for the NASA case studies defined in \fig{cases} are shown in ~\tion{dataset}. A project description file for a non-COCOMO dataset is shown in \fig{china-p2}.

\begin{figure}
\begin{verbatim}
@project china-Proj2
@attribute ?AFP vl lo md
@attribute ?Input md hi vh
@attribute ?Output vl lo md 
@attribute ?Enquiry md hi vh
@attribute ?File vl lo md
@attribute ?Interface hi vh
@attribute ?Added vl lo md
@attribute ?Changed md hi vh
@attribute ?Deleted vl lo md
@attribute ?PDR_AFP md hi vh
@attribute ?PDR_UFP md hi vh
@attribute ?NPDR_AFP vl lo md
@attribute ?NPDU_UFP vl lo md
@attribute ?Resource md hi
@attribute ?Dev.Type vl
@attribute ?Duration vl lo md hi
\end{verbatim}
\caption{Example project $controllable$ file for Chinese software projects after discretization. Ranges were assigned randomly for this project. A ``?'' represents a controllable feature. If an attribute range isn't specified in the project, it is ignored.}
\label{fig:china-p2}
\end{figure}

\section{Experiment: {\W}2 vs {\W}}\label{sect:runtimes}

Upon initial experimentation with {\W}, we often followed standard CBR methodology. For example, when deciding how to perform relevancy filtering, we chose the standard CBR practice of taking the euclidean distance from a defined project in n-dimensional space with n project features~\cite{shepperd97}. While this method performed well ~\cite{brady10b}, the $O(n^2)$ runtime requirement prevented us from practically running {\W} on very large datasets.

To resolve this, a simpler method for relevancy was devised. Instead of measuring relevancy based on the distance from a case to the project query's hypervolume, we decided to simply test for inclusion within this volume. The {\em overlap} of a case is simply the number of attributes that fall within the project query's ranges. Because our attributes must be discretized and often rely on qualitative metrics, large overlaps between a query and possible cases are common. The performance of this new method is shown in ~\fig{bestoverlap-vs-knn}. 

{\W}2 is not a slow algorithm. Nothing in any of these steps takes more than  log-linear time, and even that is only to
sort $K_1$ items (which is a very small list).
Even when implemented in an interpreted language  (GAWK), {\W}2 runs in less than half
a second for up to 500 cases (on a 3GHz dual core Macintosh, OS/X 10.6, 4GB of ram).

\begin{figure}[!t]
\begin{center}
\begin{tabular}{|r|r|r|r|r|}\cline{3-4}
 \multicolumn{2}{c}{~}& \multicolumn{2}{|c|}{Execution Time} & \multicolumn{1}{c}{~}\\\hline
dataset     & Cases & W     & W2     & W2 speedup\\\hline
telecom1    & 18        & 0.07s  & 0.04s & 1.6x\\
coc81       & 63        & 0.43s  & 0.08s & 5.3x\\
nasa93      & 93        & 0.69s  & 0.10s & 6.6x\\
china	    & 499       & 7.37s  & 0.42s & 10.8x\\\hline
\end{tabular}
\end{center}
\caption{Average execution times for the W and W2 algorithms. By removing the $O(n^2)$ kth nearest neighbor calculation from W we drastically improve performance, especially on larger datasets such as China (499 cases).}
\label{fig:w2-runtimes}
\end{figure}

\input{figs/knn-vs-bestoverlap}

In all but one case, {\W}2 performs better. However, even when {\W}2 performs slightly worse, it still performs better than KNN in spread reduction. For the Miyazaki dataset, there exists a statistically significant difference (Mann-Whitney, 95\% confidence level).

\section{Experiment: {\W}2's Performance Across Multiple Datasets}\label{sect:multi-comp}

To demonstrate the effectiveness of {\W}2 in any data environment, we offer median reductions for effort reduction for five arbitrary datasets from the PROMISE data repository. The model-agnostic simplicity of {\W}2 made implementing these tests easy as one need only describe a query space and a target utility measure. In the case of these five datasets, software effort was the common target for reduction. 

Given that we did not have access to case studies as we did with NASA93 and COC81 (ground, flight, osp, and osp2) for these datasets, synthetic queries were developed. Three queries were generated for each of the five datasets. The first contained the entire space of possible project attribute values (All), representing complete freedom to recommend any change within the space. The other two queries were generated by randomly choosing 50\% of each attribute values from either the lower, middle, or upper ranges for each project attribute (Proj1, Proj2). These queries represent more common restrictions on possible changes for a given software project.

\begin{figure}[!h]
\begin{center}
\begin{tabular}{|c|c|r|r|}\cline{3-4}
 \multicolumn{2}{c}{~}& \multicolumn{2}{|c|}{Improvement}\\\hline
dataset&query $q$&median&spread\\\hline
Telecom  & Proj1 & 96\% & 23\%\\
Telecom  & Proj2 & 91\% & 41\%\\
Telecom  & All   & 86\% & 28\%\\
Miyazaki & All   & 78\% & 33\%\\
Miyazaki & Proj2 & 69\% & 21\%\\
Miyazaki & Proj1 & 53\% & 67\%\\
Finnish  & All   & 22\% & 31\%\\
Finnish  & Proj2 & 11\% & 27\%\\
Finnish  & Proj1 & 4\% & 25\%\\
China    &  All  & 20\% & 55\%\\
China    & Proj2 & 14\% & 43\%\\
China    & Proj1 & \cellcolor[rgb]{0.8,0.8,0.8}0\% & 13\%\\
Kemerer  & Proj1 & 21\% & 61\%\\
Kemerer  & Proj2 & \cellcolor[rgb]{0.8,0.8,0.8}0\% & 49\%\\
Kemerer  & All   & \cellcolor[rgb]{0.8,0.8,0.8}-4\% & 53\%
\\\hline
\multicolumn{2}{r|}{median} & 21\% & 33\% \\\cline{3-4}
\end{tabular}
\end{center}
\caption{Effort estimation improvements ($100*\frac{initial-final}{intial}$)
for five unique datasets. Sorted by median improvement. Gray cells represent no improvement in effort estimates.}
\label{fig:other-reduce-values}
\end{figure}

\begin{figure}[!t]
\center
\includegraphics[width=5.5in]{figs/noncocomo-reduce.pdf}
\caption{Effort results for five non-COCOMO datasets.
}\label{fig:other-reduce}
\end{figure}

\begin{figure}[!b]
\center
\includegraphics[width=5.5in]{figs/changes.pdf}
\caption{Range of changes in median and spread generated by applying
the recommendations of {\W}2. The median observed
changes were (20.5, 20.5)\% for (medians, spreads), respectively. 
}\label{fig:changes}
\end{figure}


Effort reductions can be seen in ~\fig{other-reduce-values} and ~\ref{fig:other-reduce}. The chart in ~\fig{other-reduce-values} shows strong improvement in median effort for the Telecom and Miyazaki datasets, with strong performance in spread reduction across all datasets. While the Finnish, China, and Kemerer datasets show only marginal or no improvement in median effort, the certainty of their estimations is improved via a reduction in spread.

The other
two queries were generated by randomly choosing 50\% of each attribute
values from either the lower, middle, or upper ranges for each
project descriptor.
For these experiments, the $values$ function was just ``reduce effort''
(the next experiment will explore other results on COCOMO-related data, that tries to reduce effort {\em and} defects {\em and} calendar months).

There are three noteworthy aspects of the \fig{other-reduce} results:
\bi
\item All the points in that figure are positive; i.e.
improvements were seen in all cases.
\item
The dotted lines show the 50\% percentile range of the results: 
half that results had at least
56\% and 73\% improvement in median and spread.
\item
There is no evidence that {\W}2 has problems with
smaller data sets. The two smallest examples processed by {\W}2 are Kemerer and Telecom
containing 15 and 18 examples each. The minimum improvements seen, even for these small data sets,
are 55\% (in both median and spread). 
\ei
The expected value  of the results in these examples is very high; e.g. a 56\% median improvement
in effort. 
The reason for these large improvements is that, in these examples, we focuses {\em only } on effort.
Clearly, there are many ways to cut corners in a project and some of those can have disastrous
results (e.g. allocate no effort to testing will reduce the cost, but that is clearly not a recommended
management action for  a software project). 
Later in this paper are examples where 
{\W}2 is chasing improvements to effort {\em and} defects {\em and} total calendar time to
develop the software. Optimizing for $N=3$ goals is a harder task than just the $N=1$ goal of \fig{other-reduce}.
Hence, those  
the improvements seen in those examples will be more modest (around 20\%).

\section{Experiment: Intra- and Inter-Project Stability}\label{sect:stable}
One of the premises of case-based methods like {\W}2 is that local reasoning
in some specific context is best that fitting one model over an entire space.
This is required if the ``best'' solutions in a one context do not hold in others.

To test this  premise, we generated a report of what treatments were found under different
conditions.
\fig{stability} shows the results of {\W}2's $Step7$ (prune all treatments
that do appear in less than 50\% of 20 repeated trials).
\begin{figure}[!t]
\begin{center}
{\footnotesize
\begin{tabular}{|r|@{\hspace{0.75mm}}c@{\hspace{0.75mm}} @{\hspace{0.75mm}}c@{\hspace{0.75mm}} @{\hspace{0.75mm}}c@{\hspace{0.75mm}} @{\hspace{0.75mm}}c@{\hspace{0.75mm}} @{\hspace{0.75mm}}c@{\hspace{0.75mm}} @{\hspace{0.75mm}}c@{\hspace{0.75mm}} @{\hspace{0.75mm}}c@{\hspace{0.75mm}} @{\hspace{0.75mm}}c@{\hspace{0.75mm}} @{\hspace{0.75mm}}c@{\hspace{0.75mm}} @{\hspace{0.75mm}}c@{\hspace{0.75mm}} @{\hspace{0.75mm}}|c@{\hspace{0.75mm}}|c|}\hline
\multicolumn{12}{|c|}{Stability Comparison for NASA93ii FLIGHT}\\\hline
\emph{\begin{sideways}goal\end{sideways}}&
\emph{\begin{sideways}acap=3\end{sideways}}&
\emph{\begin{sideways}apex=3\end{sideways}}&
\emph{\begin{sideways}apex=5\end{sideways}}&
\emph{\begin{sideways}ltex=4\end{sideways}}&
\emph{\begin{sideways}pmat=3\end{sideways}}&
\emph{\begin{sideways}pmat=4\end{sideways}}&
\emph{\begin{sideways}rely=3\end{sideways}}&
\emph{\begin{sideways}sced=2\end{sideways}}&
\emph{\begin{sideways}stor=3\end{sideways}}&
\emph{\begin{sideways}time=3\end{sideways}}&
\emph{\begin{sideways}changes\end{sideways}}\\\hline
defects&  &  &  &  & 90 &  &  &  & 65 & 75 & 3\\
effort &  &  &  &  & 70 &  &  &  & & 90 & 2\\
months &  &  &  &  & 70 &  &  &  & 60 & 85 & 3\\
all &  &  &  &  & 75 &  &  &  & 70 & 85 & 3\\\hline
\multicolumn{12}{|c|}{Stability Comparison for NASA93ii GROUND}\\\hline
\emph{\begin{sideways}goal\end{sideways}}&
\emph{\begin{sideways}acap=3\end{sideways}}&
\emph{\begin{sideways}apex=3\end{sideways}}&
\emph{\begin{sideways}apex=5\end{sideways}}&
\emph{\begin{sideways}ltex=4\end{sideways}}&
\emph{\begin{sideways}pmat=3\end{sideways}}&
\emph{\begin{sideways}pmat=4\end{sideways}}&
\emph{\begin{sideways}rely=3\end{sideways}}&
\emph{\begin{sideways}sced=2\end{sideways}}&
\emph{\begin{sideways}stor=3\end{sideways}}&
\emph{\begin{sideways}time=3\end{sideways}}&
\emph{\begin{sideways}changes\end{sideways}}\\\hline
defects &  &  &  &  & 80 &  &  &  & 50 &  & 2\\
effort &  &  &  &  & 60 &  &  & &  & 75 & 2\\
months &  &  &  &  & 75 &  & 50 &  &  & 75 & 3\\
all &  &  &  &  & 85 &  &  &  & & 80 & 2\\\hline
\multicolumn{12}{|c|}{Stability Comparison for NASA93ii OSP}\\\hline
\emph{\begin{sideways}goal\end{sideways}}&
\emph{\begin{sideways}acap=3\end{sideways}}&
\emph{\begin{sideways}apex=3\end{sideways}}&
\emph{\begin{sideways}apex=5\end{sideways}}&
\emph{\begin{sideways}ltex=4\end{sideways}}&
\emph{\begin{sideways}pmat=3\end{sideways}}&
\emph{\begin{sideways}pmat=4\end{sideways}}&
\emph{\begin{sideways}rely=3\end{sideways}}&
\emph{\begin{sideways}sced=2\end{sideways}}&
\emph{\begin{sideways}stor=3\end{sideways}}&
\emph{\begin{sideways}time=3\end{sideways}}&
\emph{\begin{sideways}changes\end{sideways}}\\\hline
defects & 95 &  & 70 &  &  &  &  & 50 &  &  & 3\\
effort & 80 & 70 &  &  &  &  &  & 70 &  &  & 3\\
months &  &  &  &  &  & 55 &  & 80 &  &  & 2\\
all & 50 &  &  &  &  &  &  & 85 & 60 &  & 3\\\hline
\multicolumn{12}{|c|}{Stability Comparison for NASA93ii OSP2}\\\hline
\emph{\begin{sideways}goal\end{sideways}}&
\emph{\begin{sideways}acap=3\end{sideways}}&
\emph{\begin{sideways}apex=3\end{sideways}}&
\emph{\begin{sideways}apex=5\end{sideways}}&
\emph{\begin{sideways}ltex=4\end{sideways}}&
\emph{\begin{sideways}pmat=3\end{sideways}}&
\emph{\begin{sideways}pmat=4\end{sideways}}&
\emph{\begin{sideways}rely=3\end{sideways}}&
\emph{\begin{sideways}sced=2\end{sideways}}&
\emph{\begin{sideways}stor=3\end{sideways}}&
\emph{\begin{sideways}time=3\end{sideways}}&
\emph{\begin{sideways}changes\end{sideways}}\\\hline
defects &  &  &   & 60 &  & 75 &  & 95 &  &  & 3\\
effort &  &  &  & 75 &  & 55 & & 80 &  &  & 3\\
months &  &  &  & 70 &  & 80 &  & 90 &  &  & 3\\
all &  &  &  & 80 &  & 90 &  & 100 & &  & 3\\\hline
\end{tabular}}
\end{center}
\caption{Recommendation frequency across 20 runs of {\W}2 for reducing individual goals 
($defects$, $effort$, or $months$) as well as all goals at once ($all$).}
\label{fig:stability}
\end{figure}
The left-hand column of \fig{stability} shows the four $values$ function used
in that study:
\be
\item {\em Defects} aims at reducing just defects;
\item {\em Effort} aims at  reducing just effort;
\item {\em Months} aims at reducing
a project's total calendar time.
\item {\em All} refers to \eq{edt}; i.e. try to decrease effort {\em and} development time
{\em and} number of defects;
\ee
The last of these is a multi-objective function while the rest strive to optimize
one objective without concern to the others.
\fig{stability} shows that,
in any row, the conclusions reached by {\W}2 are stable (i.e. appear at high frequency,
across 20 random selections of $train:test$). 
That is, {\W}2's results exhibit
$intra-$project conclusion stability (when the
$context$ and $values$ function are held constant).
For project managers, this is good news since it shows that their data contains
clear signals on how to best change their particular project in order to achieve particular
goals.

However, \fig{stability}  also shows that  
if the $context$ is changed (from generalized FLIGHT systems to a specific
flight system like OSP), then the recommended changes are very different.
Similarly, the OSP results show that
altering the $values$ function also dramatically changes recommendations.

Menzies \& Shull~\cite{me10b} report that many SE papers conclude that
what is ``best'' for one project may not
be ``best'' for another.
For example, Zimmermann studied 629 pairs of software development 
projects~\cite{zimmermann09}. In only 4\% of those hundreds of pairs was a 
defect prediction model learned from one project useful on 
another. 
When such {\em inter-}project conclusion instability exists,
 then
tools like {\W}2 are essential since
  it is best  to learn changes that are tuned to the specifics of particular projects (like OSP \&
OSP2) rather than on generalized descriptions of software (like FLIGHT \& GROUND).

\section{Experiment: Comparing Drastic Changes to {\W}2}\label{sect:drastic-comp}

Prior work considered explored impact of so-called {\em drastic} changes to software projects~\cite{me09a}. A drastic change occurs when the recommendation falls outside the defined projects ranges of a software project. In other words, when the recommended course of action is dramatic. For example, with the OSP NASA case study (~\fig{cases}), attempting to improve programmer language and tool experience (ltex) to 5 (very high, 6+ years experience), would be a drastic change as the maximum defined value for ltex is 4 (high). 

To test this, {\W}2's recommendations for the four NASA case studies (~\fig{cases}) in the NASA93 dataset were overridden with the drastic changes from ~\fig{drastic-overview}. For 3 distinct drastic changes, {\W}2 attempted to apply the drastic changes until no improvement was measured, then reported the median effort, defects, and months for that change. Note that not all changes from ~\cite{me09a} were applicable, due to a lack of extreme cases in the NASA93 dataset.

The results for these changes are reported in Figures ~\ref{fig:drastic-flight},~\ref{fig:drastic-ground},~\ref{fig:drastic-osp},~\ref{fig:drastic-osp2}. Of the 12 comparisons, in only one case does {\W}2 perform better statistically and significantly better than the three drastic changes. However, in terms of median reductions, {\W}2 always performs in the top 50\% of of cases. Most importantly, even when compared to extreme project recommendations, {\W}2 is not constrained by the limited project attribute ranges allowed for its recommendations.

\begin{figure}[!t]
\begin{center}
\begin{tabular}{c l l}
  & \textbf{Drastic Change}   & \textbf{Attribute Effects}\\\hline
1 & Improve Process Maturity  & pmat = 5\\\hline
2 & Improve Tools\&Techniques & time = 3; stor = 3;\\
  &                           & pvol = 2; tool = 5; site = 6\\\hline
3 & Reduce Functionality      & data = 2;\\\hline
\end{tabular}
\end{center}
\caption{Examples of drastic changes to software projects.}
\label{fig:drastic-overview}
\end{figure}

\input{figs/drastic.tex}