\subsection{Comparing {\Large {\W}} to SEESAW}

In order to compare {\W} and SEESAW, both systems require similar inputs.
SEESAW can only handle models in the COCOMO format. Hence,  we restrict ourselves to data in that format
(see~\cite{me10c}
for examples of {\W} running on a much broader set of inputs). 

The inputs required for this study are:
\bi
\item {\W} needs a set of {\em historical cases}.
We used the 
NASA93 dataset available from \texttt{http://promisedata.org/data}. This dataset represents 93 different NASA projects collected from the 1980's and 1990's represented as feature vectors describing each project in COCOMO format.
NASA93 data only contains historical information for
project effort. Development time (measured in calendar months) and defects were added in using the COCOMO/COQUALMO models.
\item Both SEESAW and {\W} need an {\em objective function} that guides their search. 
In this study, the objective function rewarded
minimization of the sum of defects and effort and months (after these values had been normalized
to the same range).
\item Both SEESAW and {\W} need a set of {\em project constraints} that tune their conclusions to particular projects. We used the project constraints of \fig{cases}.
\ei
\fig{cases} comes from our debriefing of NASA program managers and shows
different
kinds of NASA mission:
\bi
\item \emph{Ground} and \emph{flight} represent typical ranges for most NASA projects at the Jet Propulsion Laboratory (JPL);
\item \emph{OSP} represents the guidance, navigation, and control aspects of NASA's 1990 Orbital Space Plane;
\item \emph{OSP2} represents a second, later version of OSP with a more limited scope of COCOMO attributes.
\ei
The $values$ column in that figure shows settings that cannot be
changed; e.g. for OSP, the required reliability is fixed at $rely=5$.
On the other hand, the $low$ and $high$ ranges in that figure define
the space of possible recommendations for that project. For instance,
the reliability of the JPL flight software can vary from a ranking
of 3 (nominal) to 5 (very high). 

{\W} used \fig{cases} to set its initial query $q_0$. 
SEESAW used \fig{cases} to guide a set of simulations around its parametric models. For each case
study, 1000 times, inputs were selected
at random, constrained by \fig{cases} (so the inputs 
for case study $X$ conformed to the description of $X$ shown in that figure).

\input{figs/projects.tex}

In order to offer a fair comparison between SEESAW and {\W}, we proceeded as follows.
Recall that {\W} has a {\em training} component that implements
 {\em retrieve}, {\em reuse}, and {\em revise} (described around \fig{w}).
A {\em test} component was implemented that copied the code used for
{\em retrieve}. This test component was modified such that it executed
on a different {\em test set} that contained no data used in {\em training}.

Given that rig, for each case study in \fig{cases}, we repeated the following process 50 times.
\bi
\item
The available data (NASA93)
was divided into a {\em train} and {\em test} sets (of sizes  66\%:33\%).
The division was random so that each time, different instances appeared in train and test.
\item
The median and spread values for effort, months, and defects were collected from the
{\em train set}. 
These medians and spreads were  recorded as  the $before$ values.
\item
Each quality optimizer ({\W} and SEESAW) was run separately.
The {\W} algorithm used the {\em train set} while SEESAW used its internal models.
In either case, the quality optimizer returned a set of recommendations on how to change the project
in order to reduce effort, defects, and development time (measured in calendar months).
\item
These recommendation were assessed in the same way: by
 passing them to {\W}'s test component which retrieved  relevant cases from the {\em test set}.
\item
The median and spread values for effort, months, and defects were collected from the
instances retrieved from the {\em test set}. These were recorded as  the $after$ values.
\ei
The results were reported in terms of {\em median} and {\em spread}.
We say that the $median$ of a set of numbers are the 50th-percentile value while
the $spread$ is difference between the 75th and 25th percentile value.
The median is a measure of central tenancy while the spread
is a measure of uncertainty around the median.
Decreasing the spread means that the predictions
fall within a narrower range. We report spread rather than other measures like standard deviation
since we wish to avoid any inappropriate assumptions of symmetrical distributions.

\subsection{Comparing {\Large {\W}} to {\Large {\W}}2}

\subsubsection{Effectiveness}

Upon initial experimentation with {\W}, we often followed standard CBR methodology. For example, when deciding how to perform relevancy filtering, we chose the standard CBR practice of taking the euclidean distance from a defined project in n-dimensional space with n project features~\cite{shepperd97}. While this method performed well ~\cite{10promisew}, the $O(n^2)$ runtime requirement prevented us from practically running {\W} on very large datasets.

To resolve this, a simpler method for relevancy was devised. Instead of measuring relevancy based on the distance from a case to the project query's hypervolume, we decided to simply test for inclusion within this volume. The {\em overlap} of a case is simply the number of attributes that fall within the project query's ranges. Because our attributes must be discretized and often rely on qualitative metrics, large overlaps between a query and possible cases are common.

The performance of this new method is shown in ~\fig{bestoverlap-vs-knn}. In all but one case, {\W}2 performs better. However, even when {\W}2 performs slightly worse, it still performs better than KNN in spread reduction. For the Miyazaki dataset, there exists a statistically significant difference (Mann-Whitney, 95\% confidence level).

\subsubsection{Execution Speed}

\fig{w2-runtimes} shows the average execution times for {\W} and {\W}2 across four datasets. While the original {\W}'s simplicity affords it runtimes of less than one second even on datasets with 93 instances (NASA93), the $O(n^2)$ kth nearest neighbor calculation becomes more significant when dealing with datasets as large as China with 500 instances. By replacing the $O(n^2)$ knn calculation with the linear {\em overlap} calculation, we maintain runtimes of less than half a second for larger datasets.

Runtimes of {\W}2
\input{figs/w2-runtimes.tex}

\input{figs/knn-vs-bestoverlap.tex}

\subsubsection{Stability}

An unexpected cause for concern with {\W} was some instability in its recommendations. After removing the nearest neighbor filtering, the {\em overlap} method for case relevancy filtering helps reduce this affect, as seen in ~\fig{stability}. Across four different goals (reducing defects only, effort only, months only, or all at once) {\W}2 returns more stable improvements, whereas {\W} tends to shift it's recommendations when changing goals.

\begin{figure}[!h]
\begin{center}
{\scriptsize
\begin{tabular}{|c|@{\hspace{0.5mm}}c@{\hspace{0.5mm}}|@{\hspace{0.5mm}}c@{\hspace{0.5mm}} @{\hspace{0.5mm}}c@{\hspace{0.5mm}} @{\hspace{0.5mm}}c@{\hspace{0.5mm}} @{\hspace{0.5mm}}c@{\hspace{0.5mm}} @{\hspace{0.5mm}}c@{\hspace{0.5mm}} @{\hspace{0.5mm}}c@{\hspace{0.5mm}} @{\hspace{0.5mm}}c@{\hspace{0.5mm}} @{\hspace{0.5mm}}c@{\hspace{0.5mm}} @{\hspace{0.5mm}}c@{\hspace{0.5mm}} @{\hspace{0.5mm}}c@{\hspace{0.5mm}} @{\hspace{0.5mm}}c@{\hspace{0.5mm}} @{\hspace{0.5mm}}c@{\hspace{0.5mm}} @{\hspace{0.5mm}}c@{\hspace{0.5mm}} @{\hspace{0.5mm}}c@{\hspace{0.5mm}} @{\hspace{0.5mm}}c@{\hspace{0.5mm}} @{\hspace{0.5mm}}c@{\hspace{0.5mm}} @{\hspace{0.5mm}}c@{\hspace{0.5mm}} @{\hspace{0.5mm}}c@{\hspace{0.5mm}}|c|}\hline
\multicolumn{21}{|c|}{Stability Comparision for NASA93 FLIGHT}\\\hline
\emph{\begin{sideways}query\end{sideways}}&
\emph{\begin{sideways}goal\end{sideways}}&
\emph{\begin{sideways}acap=3\end{sideways}}&
\emph{\begin{sideways}apex=3\end{sideways}}&
\emph{\begin{sideways}apex=5\end{sideways}}&
\emph{\begin{sideways}cplx=4\end{sideways}}&
\emph{\begin{sideways}data=2\end{sideways}}&
\emph{\begin{sideways}data=3\end{sideways}}&
\emph{\begin{sideways}ltex=3\end{sideways}}&
\emph{\begin{sideways}ltex=4\end{sideways}}&
\emph{\begin{sideways}plex=3\end{sideways}}&
\emph{\begin{sideways}pmat=2\end{sideways}}&
\emph{\begin{sideways}pmat=3\end{sideways}}&
\emph{\begin{sideways}pmat=4\end{sideways}}&
\emph{\begin{sideways}rely=4\end{sideways}}&
\emph{\begin{sideways}rely=4\end{sideways}}&
\emph{\begin{sideways}sced=2\end{sideways}}&
\emph{\begin{sideways}sced=3\end{sideways}}&
\emph{\begin{sideways}stor=3\end{sideways}}&
\emph{\begin{sideways}time=3\end{sideways}}&
\emph{\begin{sideways}changes\end{sideways}}\\\hline
\multirow{4}{*}{\begin{sideways}W2\end{sideways}} & d &  &  &  &  &  &  &  &  &  &  & 90 &  &  &  &  &  & 65 & 75 & 3\\
 & e &  &  &  &  &  &  &  &  &  &  & 70 &  &  &  &  &  &  & 90 & 2\\
 & m &  &  &  &  &  &  &  &  &  &  & 70 &  &  &  &  &  & 60 & 85 & 3\\
 & dem &  &  &  &  &  &  &  &  &  &  & 75 &  &  &  &  &  & 70 & 85 & 3\\\hline
\multirow{4}{*}{\begin{sideways}W\end{sideways}} & dem & 55 &  &  &  &  &  &  &  &  &  &  &  &  &  &  &  &  & 50 & 2\\
 & d &  &  &  & 60 &  &  &  &  & 55 &  &  &  &  &  &  &  &  &  & 2\\
 & e &  &  &  &  &  &  &  &  &  &  &  &  &  &  &  &  & 70 & 70 & 2\\
 & m &  &  &  &  & 55 &  &  &  &  &  &  &  &  &  &  &  &  &  & 1\\\hline

\multicolumn{21}{|c|}{Stability Comparision for NASA93 OSP2}\\\hline
\emph{\begin{sideways}query\end{sideways}}&
\emph{\begin{sideways}goal\end{sideways}}&
\emph{\begin{sideways}acap=3\end{sideways}}&
\emph{\begin{sideways}apex=3\end{sideways}}&
\emph{\begin{sideways}apex=5\end{sideways}}&
\emph{\begin{sideways}cplx=4\end{sideways}}&
\emph{\begin{sideways}data=2\end{sideways}}&
\emph{\begin{sideways}data=3\end{sideways}}&
\emph{\begin{sideways}ltex=3\end{sideways}}&
\emph{\begin{sideways}ltex=4\end{sideways}}&
\emph{\begin{sideways}plex=3\end{sideways}}&
\emph{\begin{sideways}pmat=2\end{sideways}}&
\emph{\begin{sideways}pmat=3\end{sideways}}&
\emph{\begin{sideways}pmat=4\end{sideways}}&
\emph{\begin{sideways}rely=4\end{sideways}}&
\emph{\begin{sideways}rely=4\end{sideways}}&
\emph{\begin{sideways}sced=2\end{sideways}}&
\emph{\begin{sideways}sced=3\end{sideways}}&
\emph{\begin{sideways}stor=3\end{sideways}}&
\emph{\begin{sideways}time=3\end{sideways}}&
\emph{\begin{sideways}changes\end{sideways}}\\\hline
\multirow{4}{*}{\begin{sideways}W2\end{sideways}} & d &  &  &  &  &  &  &  & 60 &  &  &  & 75 &  &  & 95 &  &  &  & 3\\
 & e &  &  &  &  &  &  &  & 75 &  &  &  & 55 &  &  & 80 &  &  &  & 3\\
 & m &  &  &  &  &  &  &  & 70 &  &  &  & 80 &  &  & 90 &  &  &  & 3\\
 & dem &  &  &  &  &  &  &  & 80 &  &  &  & 90 &  &  & 100 &  &  &  & 3\\\hline
\multirow{4}{*}{\begin{sideways}W\end{sideways}} & dem &  &  &  &  &  &  & 65 &  &  &  & 95 &  &  &  &  & 75 &  &  & 3\\
 & d &  &  &  &  &  &  & 50 &  &  &  & 80 &  &  &  &  & 100 &  &  & 3\\
 & e &  &  &  &  &  &  & 60 &  &  & 80 &  &  &  &  &  & 75 &  &  & 3\\
 & m &  &  &  &  &  &  & 65 &  &  &  & 95 &  &  &  &  & 95 &  &  & 3\\\hline

\multicolumn{21}{|c|}{Stability Comparision for NASA93 GROUND}\\\hline
\emph{\begin{sideways}query\end{sideways}}&
\emph{\begin{sideways}goal\end{sideways}}&
\emph{\begin{sideways}acap=3\end{sideways}}&
\emph{\begin{sideways}apex=3\end{sideways}}&
\emph{\begin{sideways}apex=5\end{sideways}}&
\emph{\begin{sideways}cplx=4\end{sideways}}&
\emph{\begin{sideways}data=2\end{sideways}}&
\emph{\begin{sideways}data=3\end{sideways}}&
\emph{\begin{sideways}ltex=3\end{sideways}}&
\emph{\begin{sideways}ltex=4\end{sideways}}&
\emph{\begin{sideways}plex=3\end{sideways}}&
\emph{\begin{sideways}pmat=2\end{sideways}}&
\emph{\begin{sideways}pmat=3\end{sideways}}&
\emph{\begin{sideways}pmat=4\end{sideways}}&
\emph{\begin{sideways}rely=4\end{sideways}}&
\emph{\begin{sideways}rely=4\end{sideways}}&
\emph{\begin{sideways}sced=2\end{sideways}}&
\emph{\begin{sideways}sced=3\end{sideways}}&
\emph{\begin{sideways}stor=3\end{sideways}}&
\emph{\begin{sideways}time=3\end{sideways}}&
\emph{\begin{sideways}changes\end{sideways}}\\\hline
\multirow{4}{*}{\begin{sideways}W2\end{sideways}} & d &  &  &  &  &  &  &  &  &  &  & 80 &  &  &  &  &  & 50 &  & 2\\
 & e &  &  &  &  &  &  &  &  &  &  & 60 &  &  &  &  &  &  & 75 & 2\\
 & m &  &  &  &  &  &  &  &  &  &  & 75 &  & 50 &  &  &  &  & 75 & 3\\
 & dem &  &  &  &  &  &  &  &  &  &  & 85 &  &  &  &  &  &  & 80 & 2\\\hline
\multirow{4}{*}{\begin{sideways}W\end{sideways}} & dem &  &  &  &  &  &  &  &  &  &  &  &  &  &  &  &  &  & 50 & 1\\
 & d &  &  &  &  & 70 &  &  &  &  &  &  &  &  &  &  &  &  &  & 1\\
 & e &  &  &  &  &  & 60 &  &  & 55 &  &  &  &  &  &  &  & 50 &  & 3\\
 & m &  &  &  &  &  &  &  &  &  &  &  &  &  & 65 &  &  &  &  & 1\\\hline

\multicolumn{21}{|c|}{Stability Comparision for NASA93 OSP}\\\hline
\emph{\begin{sideways}query\end{sideways}}&
\emph{\begin{sideways}goal\end{sideways}}&
\emph{\begin{sideways}acap=3\end{sideways}}&
\emph{\begin{sideways}apex=3\end{sideways}}&
\emph{\begin{sideways}apex=5\end{sideways}}&
\emph{\begin{sideways}cplx=4\end{sideways}}&
\emph{\begin{sideways}data=2\end{sideways}}&
\emph{\begin{sideways}data=3\end{sideways}}&
\emph{\begin{sideways}ltex=3\end{sideways}}&
\emph{\begin{sideways}ltex=4\end{sideways}}&
\emph{\begin{sideways}plex=3\end{sideways}}&
\emph{\begin{sideways}pmat=2\end{sideways}}&
\emph{\begin{sideways}pmat=3\end{sideways}}&
\emph{\begin{sideways}pmat=4\end{sideways}}&
\emph{\begin{sideways}rely=4\end{sideways}}&
\emph{\begin{sideways}rely=4\end{sideways}}&
\emph{\begin{sideways}sced=2\end{sideways}}&
\emph{\begin{sideways}sced=3\end{sideways}}&
\emph{\begin{sideways}stor=3\end{sideways}}&
\emph{\begin{sideways}time=3\end{sideways}}&
\emph{\begin{sideways}changes\end{sideways}}\\\hline
\multirow{4}{*}{\begin{sideways}W2\end{sideways}} & d & 95 &  & 70 &  &  &  &  &  &  &  &  &  &  &  & 50 &  &  &  & 3\\
 & e & 80 & 70 &  &  &  &  &  &  &  &  &  &  &  &  & 70 &  &  &  & 3\\
 & m &  &  &  &  &  &  &  &  &  &  &  & 55 &  &  & 80 &  &  &  & 2\\
 & dem & 50 &  &  &  &  &  &  &  &  &  &  &  &  &  & 85 &  & 60 &  & 3\\\hline
\multirow{4}{*}{\begin{sideways}W\end{sideways}} & dem &  &  &  &  &  &  &  & 95 &  &  &  &  &  &  & 90 &  &  &  & 2\\
 & d &  & 55 &  &  &  &  &  &  &  &  &  &  &  &  & 65 &  &  &  & 2\\
 & e &  & 70 &  &  &  &  &  &  &  &  &  &  &  &  & 60 &  &  &  & 2\\
 & m &  &  &  &  &  &  &  & 65 &  &  &  & 75 &  &  & 95 &  &  &  & 3\\\hline
\end{tabular}}
\end{center}
\caption{Recommendation frequency across 20 runs of {\W} and {\W}2 for reducing individual goals (defects, effort, months) as well as all goals at once (d/e/m). Recommendations that appeared less than half the time are omitted. Note that despite chaning goal functions, {\W}2 continues to make similar recommendations.}
\label{fig:stability}
\end{figure}
