\section{When Not to Use {\W}2}\label{sect:delta}
Like any case-based method, {\W}2 requires historical
cases. If such data is missing then {\W}2 cannot be used.

In that circumstance,
discussions about how to best change a project can use
results borrowed from other sites. For example, \fig{coceffects} show's Boehm et al.'s~\cite{boehm00b} 
analysis of the effects of changing some project attribute from its minimum
to maximum value. Based on 
data from
a regression analysis of 161 projects, this figure comments that
changing  (e.g.) personnel/team capability   can alter the effort
to build software by up to 350\%.  Using this data, the effects
of various changes can be investigated using Boehm's {\em delta} analysis technique~\cite{boehm00a}:
\bi
\item
An 
old project with known efforts is used as a baseline. A change
to a project is described as a new project, expressed in terms of deltas to 
the variables of \fig{coceffects}.
\item
The new estimate is then the product 
of the baseline times the effort multiplier deltas. 
\item
The ``best'' changes to a project are those that are simplest
to implement and have most positive impact on the effort (ideally, reduces it).
\ei
Column two of
\fig{coceffects} lets us compare context-independent reasoning
(e.g. delta analysis) vs. context-dependent reasoning (e.g. {\W}2).  
Note  how that only a third of the \fig{coceffects} attributes appear in the ``best''
treatments of \fig{stability}. 
Curiously, the two
attributes with greatest impact (personnel/team capability and product complexity) are absent from
\fig{stability}. 

Why is {\W}2 ignoring an attribute with such a large impact (350\%)?
To answer that question, we have go to the context-dependent particulars.
Recall from \fig{cases} that in OSP2, product complexity is fixed at $cplx=4$ and personnel/team capability is
fixed at $pcap=3$.
{\W}2 does not recommend treatments for
things that cannot change. Hence, $cplx$ and $pcap$ are absent from the OSP2 results of \fig{stability}.
Similarly, OSP allows only $pcap=3$ so this attribute is also absent.

The same reasoning does not explain the other absent attributes.
To understand these,  we must look at the data.
OSP sets $cplx\in\{5,6\}$. This attribute is absent in the treatments since there is insufficient
support in NASA93ii to justify their inclusion (there  only five
$cplx=5$ examples in NASA93ii and no examples of $cplx=6$).
Similar explanations can explain all the remaining absences.
\begin{figure}[!t]
\footnotesize
\begin{center}
\begin{tabular}{r@{~}|c|@{~}l@{~}r@{~}l}
%& & relative\\
   & appears in \\
id &\fig{stability} as& features  & \multicolumn{2}{l}{relative weight}\\\hline
1&&Personnel/team capability & 3.53	&\rule{17mm}{2mm}\\
2&&  Product complexity & 2.38&	\rule{12mm}{2mm}\\
3&time& Time constraint & 1.63&	\rule{8mm}{2mm}\\
4&rely&    Required software reliability & 1.54&	\rule{8mm}{2mm}\\
5&&    Multi-site development   &	1.53&	\rule{7mm}{2mm}\\
6&&       Doc. match to life cycle  & 	1.52	&\rule{7mm}{2mm}\\
7&&  Personnel continuity  &	1.51&	\rule{7mm}{2mm}\\
8&apex&  Applications experience & 1.51	&\rule{7mm}{2mm}\\
9&& Use of software tools & 1.50	&\rule{7mm}{2mm}\\
10&& Platform volatility & 1.49&	\rule{7mm}{2mm}\\
11&stor& Storage constraint & 1.46&	\rule{7mm}{2mm}\\
12&pmat& Process maturity  &	1.43	&\rule{7mm}{2mm}\\
13&ltex&    Language \& tools experience  &	1.43	&\rule{7mm}{2mm}\\
14&sced&     Required dev. schedule  & 1.43	&\rule{7mm}{2mm}\\
15&&Data base size & 1.42&	\rule{7mm}{2mm}\\
16&&  Platform experience & 1.40&	\rule{7mm}{2mm}\\
17&&     Arch. \& risk resolution  &	1.39&	\rule{7mm}{2mm}\\
18&&  Precedentedness  &	1.33&	\rule{6mm}{2mm}\\
19&&  Developed for reuse  	&1.31	&\rule{6mm}{2mm}\\
20&& Team cohesion  &	1.29&	\rule{6mm}{2mm}\\
21&&    Development mode   &	1.32	&\rule{6mm}{2mm}\\
22&& Development flexibility &	1.26	&\rule{6mm}{2mm}\\
\end{tabular}
\end{center}
\caption{Relative effects  on development effort. From~\cite{boehm00a}.
}\label{fig:coceffects}
\end{figure}
Examples such as these show how {\W}2 can provide recommendations that may go against common expert advice. This lack of a defined relationship between data attributes underscores the need for careful query construction. For example, if a query contains conflicting attributes, {\W}2 maintains no internal inconsistency check. Model-based approaches such as S-COST ~\cite{Boehm96softwarecost} can provide this sanity check, but incur the
costs associated with model-based methods discussed above
(ontology restrictions, untamed internal model variance, etc).

In summary, when data is absent, managers can debate changes to projects by reusing data like \fig{coceffects}.
However, the conclusions reached from a context-independent reasoning (like delta analysis)
can be made more specific  with local information about the kinds of projects seen in the local environment
and the kinds of changes the local managers are willing to accept. Therefore, where possible, we recommend
collecting local data and analyzing it with {\W}2.

\section{Model-lite}\label{sect:modellite}
We said above that CBR was {\em model-lite}, but not {\em model-free}.
We hesitate to call CBR {\em model-free}, lest we incur the wrath
of Janet Kolodner or Roger Shank~\cite{shank77}.
Kolodner and Shank regard CBR as a {\em model} of human cognition
where knowledge in a context-dependent
manner, according to the task at hand. This construct may differ from context to context
but the search mechanisms by which the construct is built (CBR) is constant.

To expand on that point, we note that ``model'' has at least two
definitions:
\be
\item A hypothetical description of a complex entity or process. 
\item A plan to create, according to a model or models. 
\ee 
The first definition is closest to Shepperd's definition of ``model-based systems''. According to Shepperd~\cite{shepperd07} software effort estimation methods separate into ``human-centric'' techniques and ``model-based'' techniques. In the former, humans produce their recommendations without using some externalizable representation. In the latter, a variety of techniques may be used which, according to Shepperd, divide into algorithmic/parametric models (like COCOMO) and induced prediction systems (which include regression, rule induction, CBR, and many others). 

We can marry Shepperd's view with that of Kolodner and Shank by specializing the definition
of model-based systems. Extending Shepperd's ontology, we say that model-based systems can be sorted according to how much modeling they assume prior to induction. At one end of that sort order, we have parametric models like COCOMO. We call these {\em model-heavy} since they conform to the first definition of ``model'', shown above. At the other end of that sort are  the {\em model-lite} methods like CBR. These model-lite methods conform to the second definition of ``model''. Note that this second definition is closest to Kolodner and Shank's view on CBR; i.e. the CBR model is a recipe for generating context-dependent knowledge.

\section{Scope of the Study}
%\subsubsection{External Validity}

This study use conveniently available datasets in the PROMISE repository, the result applies  within the same context of the datasets. In addition,
our evaluation compares the performance of different methods
across a finite number of problems, so it  cannot be used to predict which
method will be superior to others for some future, as yet unseen, problem. 
In fact, no method has been found so far that is universally superior to others
in all problems; indeed, the ``no-free-lunch
theorem''~\cite{wolpert94} suggests  that such an universal best method for all problems can never
exist. In practice, for a given new learning problem,
various methods need to be empirically evaluated to find the best
ones, such as the ones carried out in the study.

We have shown that some treatments identified can improve the quality measures observed in historical project datasets. Our performance measures including median and spread reductions seen in ``hold-out data'' should not be confused with practical significance in the real world. 

That being said,
we note that publications from other research communities assess their models in the same manner
 as this paper:
see the {\em effort estimation}~\cite{Li2009,Lipowezky1998,Walkerden1999,Kirsopp2002,Mendes2003}
and {\em defect prediction}~\cite{ostrand04,journals/ese/KhoshgoftaarS03,tosun10,me10a} literature.
Ideally, researchers in effort estimation, defect prediction, or learning changes to software projects
should apply their recommendations to live projects. However, hold-out tests
are widely used due to the tremendous practical
difficulties associated with performing such tests on multiple software projects.
At the very least, studies like this
paper are required to prune the space of methods to be laboriously tested on new, real-world, projects.