\section{Motivation}

There are many  ways a manager might  {\em change}, and hopefully {\em improve},  their software development project. Some changes  require {\em tools} such as  using the new generation of functional programming languages or execution and testing tools~\cite{andrews07} or automated formal analysis~\cite{holzmann97}. Other changes use {\em process improvement} techniques such as changing the organizational hiring practices, or a continual renegotiation of the requirements as part of an agile software development cycle~\cite{port08}.

Endres \& Rombach~\cite{endres03} list dozens of  {\em laws} of software engineering to justify a particular change to a project. If a manager proposed using {\em all} the laws, then senior management would most likely suggest they scale back their plans to just {\em a minimal set of most effective measures}.  

Such analysis is critical early in a project's lifetime. Boehm's first law states that errors are most frequent in the design and requirements phases of development. Such errors are also more costly the later they are removed~\cite{endres03}. Software managers need tools that can suggest changes to a project given high-level characterizations early and effectively.

This thesis explores different ways  for finding this  minimal set of most effective changes to a project.
Specifically, comparing {\em model-based} vs {\em case-based} methods. The difference between these two methods is as follows.
Model-based methods develop a model via expert advice~\cite{fenton07}  or using automatic methods such as {\em data mining}~\cite{witten05}. Once built, the model can be used for ``what-if'' queries in order to assess possible changes to a project. For example: 
{\small \[
\begin{array}{rcl}
data&\rightarrow& model\\
model + whatIf& \rightarrow& scores
\end{array}
\]}
Here, $Scores$ represents business concerns; for example reduce defects before release the software product.
Also, the  $whatIf$ query defines a $context$ within which a manager seeks
ways to improve a project.

Case-based methods, on the other hand, insert the ``what-if'' query into a $n$-dimensional space populated with 
historical project cases~\cite{norvig03,She02,leake96,kol93,leake05}.
Unlike model-based methods, 
case-based methods do not require an underlying model.
Rather, the  immediate neighborhood of the ``what-if''  
is somehow scored to find summary of those neighboring  cases: 

{\small \[
\begin{array}{rcl}
data + whatIf& \rightarrow& neighborhood\\
neighborhood& \rightarrow& scores
\end{array}
\]}


Previous work~\cite{me00f,me07g,me07f,orrego09,me09a,me09e,me09f,me09i,me10f}
tried combining model-based  methods with AI tools to control  thousands of  ``What-If'' queries over COCOMO models. 
This thesis  compares those model-based methods with ``{\W}2'', a novel
case-based method.
Given a ``What-If'' query that selects some set of similar projects,
{\W}2 seeks a $treatment$ $R_x$, which finds the ``better'' parts of those
similar projects within the dataset:

{\small \[
\begin{array}{r@{~}rcl}
&data + WhatIf & \rightarrow& neighborhood_1\\
&neighborhood_1& \rightarrow& scores_1\\
R_x \;+&data + WhatIf & \rightarrow  & neighborhood_2\\
&neighborhood_2& \rightarrow& scores_2\\
&scores_2& >& scores_1  
\end{array}
\]}

\noindent
When compared to model-based methods:
\bi
\item
{\W}2 identified {\em similar} or {\em better} treatments.
\item
{\W}2 {\em was faster  to run:} all the experiments in this thesis require 10 minutes with {\W}2, but days
for using models.
\item
{\W}2 {\em was  simpler to implement:} {\W}2's  200 lines AWK replaces thousands of lines of 
the model-based LISP.
\item
{\W}2 {\em was simpler to maintain:} with case-based methods, ``maintenance'' implies ``adding more cases''.
\item 
{\W}2 {\em was simpler to adapt to new domains:}
{\W}2 do not require an underlying model and therefore it imposes no restrictions
on the data being processed. It is more efficient and it can be quickly applied to more data sets. 
\ei
Hence, this thesis recommends case-based methods like {\W}2 for identifying changes to software projects.
\indent


This result shows that the {\W}2 case-based method is superior to all the model-based methods explored in this study.
This does not imply that learning changes to software projects is {\em always} best achieved
using simple case-based methods. For example, the next release planning problem discussed in~\cite{ruhe09,zhang07a}
is a process problem of great complexity.
For that task,
the Pareto frontier optimization methods employed by (e.g.) Ruhe~\cite{ruhe09}
is preferred to {\W}2.

Therefore, this thesis offers no \emph{conclusion} on the relative merits of model-based versus case-based methods. Rather, it aims to show the relative benefits and performance potential of such case-based methods.

\section{Statement of Thesis}

{\W}2 represents a useful departure from standard model-based approaches for recommending changes to software projects. These experiments show that {\W}2 performs as well or better than more complex methods such as SEESAW, but offers greater flexibility in dataset applicability. 

While this thesis cannot offer a conclusion on the whether to use model-based or case-based methods for improving software projects, {\W}2 provides a case-study in the usefulness of case-based methods.

\section{Contribution of This Work}

The work with {\W}2 and this thesis has 4 significant contributions:
\be
\item 
{\W}2 enhances algorithms proposed by other researchers. Prior work on case-based methods found ways to $generate$ estimates~\cite{shepperd97,Mendes2003,Kirsopp2002,Walkerden1999}. {\W}2 shows that a small modification to standard case-based analysis can determine how to $change$ an estimate. 
\item
{\W}2 outperforms the model-based methods described in~\cite{me00f,me07g,me07f,orrego09,me09a,me09e,me09f,me09i,me10f}
as well as earlier versions of {\W} published in~\cite{brady10a} and~\cite{brady10b}.
As discussed in \tion{inst}, the {\W}2 algorithm presented here handles missing values better than early versions.
\item
Recommendations from {\W}2 offer competing alternatives to standard ``drastic'' management recommendations.
\item
Results show simple case-based methods can perform better than more complex model-based methods.
\ee
While the first points may be of most interest to industrial practitioners, but it is the last point  that may be most interesting to researchers. There are many sophisticated methods for exploring the complexities and uncertainties of 
trying to control software engineering projects. The results presented herein advise researchers to first explore simpler methods, if only  for the purposes of establishing a performance baseline.

\section{Document Structure}

The following chapter lays out the related work when discussing model-based and case-based methods for improving software quality. Each approach is defined along with general benefits and drawbacks. Chapter 3 formally introduces the {\W}2 algorithm and its improvements, along with a worked example of recommending a change to a given project. Chapter 4 consists of the following experiments:

\begin{itemize}
\item A comparison between the original version of {\W} and {\W}2. {\W}2 removes a $O(n^2)$ euclidean distance comparison for a linear time $overlap$ calculation while retaining similar or better performance.
\item {\W}2's effectiveness in reducing estimated software effort across a range of arbitrary datasets
\item A comparison of the decisions learned with {\W}2 to common drastic management decisions
\item The stability of the conclusions recommended by {\W}2
\end{itemize}

Chapter 5 describes the experiment between model-based and case-based algorithms for software quality optimization. SEESAW is introduced as the best-performing straw-man example of model-based algorithms. {\W}2 is then compared to SEESAW in terms of estimated reductions in software effort, defects, and development months.

Chapter 6 continues the discussion of the comparison between model-based and case-based approaches. This chapter also discusses the application of {\W}2, including when it is useful and when other approaches may be more useful. Finally, Chapter 6 states concludes one the greater implications of tools like {\W}2 and the larger choice of how to recommend changes to software projects.