Advocates of reconstructive memory such as Barlett~\cite{bartlett32}, Kolodner~\cite{kolodner83}, or Shank~\cite{shank77} argue that {\em we make it up as we go along}. In case-based reasoning (CBR), inference repeats every time there is a new query. Our reading of the papers at this conference is that, except for a few papers that deal with reasoning-by-analogy (e.g.~\cite{azzeh08}), most of this community avoids the model-lite approach of CBR.

Proponents of parametric models argue that there exist {\em domain-independent models} which can be {\em tuned} to local details.
In
this approach, reasoning can take the form of a data miner learning
values for tune-able attributes of a parametric  model like \eq{one}. In this way, learning can happen once
and users can use the tuned model for all future queries.  
%While few exceptions~\cite{weyuker07,tosun09,me09b,me10a,me10b},
%the literature does not support the existence of 
%domain-independent SE models:
%\bi
%\item Zimmerman et al.~\cite{zimmermann09}  studied 629 pairs of projects. In only 4\% of these pairs
%could a defect 
%prediction model learned from one project be applied to another.
%\item
%A recent survey by Kitchenham et al.~\cite{kitch07}  reviewed claims that effort estimation data can be usefully
%gathered in one company, then applied in another.
%They found the existing evidence inconclusive and even contradictory. 
%\ei
%In order to check if the above papers are isolated reports, or part of a more general pattern, we 
%reviewed all presentations made at PROMISE.
%Since 2005, at the PROMISE conference: 
%\bi
%\item
%There have been 68 presentations, 48 of which either tried a new analysis on old data or made 
%reports  that a new method worked for one particular project. 
%\item
%Nine papers raised questions about the validity of prior results (e.g.~\cite{me09f});
%\item Four papers argued that SE model generality was unlikely and/or impossible (e.g.~\cite{briand06});
%\item Only rarely (7/68) did researchers report generalizations from one project to other projects: 
%\bi
%\item Four papers reported a software quality predictor learned from one project that was usefully 
%applied to a new project (e.g.~\cite{tosun09,weyuker07});
%\item Three other papers made a partial case that such generality was possible (e.g.~\cite{boehm09}).
%\ei
%\ei

Unfortunately, these supposedly domain-independent models (like COCOMO) suffer from massive internal variance (see \fig{coc1}).
Previously, we have tried to manage internal variance of  this problem with SEESAW: an AI algorithm that sought stable conclusions
across the space of possible tunings within a parametric model. While a successful prototype, SEESAW has
disadvantages:
\bi
\item Dependency on  a particular parametric model
\item A requirement that all the data be in a format acceptable to that model
\item Too many arbitrary internal design decisions
\item Slow runtimes
\item A code base that proved too large to maintain, modify, and add support for more models
\ei
With a result supporting CBR, this paper finds little to recommend from SEESAW over the {\W} case-based reasoning tool. Standard CBR applies a query $q$ to find relevant examples from a set of cases $C$ using the retrieve-reuse-revise-retain loop of \fig{cbr}. {\W} extends standard CBR by learning an adaption of $q$, called $q'$, that retrieves {\em better} quality examples. Based on the analysis of~\cite{me10c} and this paper, we recommend {\W} on several grounds:
\bi
\item
{\W} finds similar, or better, results than SEESAW (see \fig{win-loss-tie}).
\item
{\W}  is simpler to code:
200 lines of AWK as opposed to the 5000 lines
of LISP code used in SEESAW.
\item
{\W} is
faster to run: the above experiments took minutes for {\W},
but hours for SEESAW.
\item
{\W} is simpler to maintain since, in CBR, ``maintenance'' means nothing more than  ``add more cases''.
\item
{\W} makes no use of an underlying
model and is therefore free from the assumptions
of parametric modeling. Hence it can be applied to more data sets. For example,
SEESAW requires data to be in the COCOMO format but {\W} has been applied to numerous
data sets in other formats~\cite{me10c}.
\ei
Having said that, there is one situation where we'd recommend SEESAW over {\W}. Like all CBR systems,
{\W} needs cases. If there is {\em no} local data, then
SEESAW would be the preferred (only) option. 

What then should we say about the premise of PROMISE; i.e. that ``modeling'' is an appropriate method for understanding SE projects? Our answer is two-fold. Firstly, there is insufficient evidence in this paper to make the conclusion that CBR {\em always} beats model-heavy methods like parametric models.
Neverthless,  these results clearly motivate further exploration and comparison between the value of CBR and model-heavy techniques. For example, at our lab we are exploring very fast clustering methods to support scaling CBR to very large data sets.

Secondly, there are at least two kinds of ``models.'' In the traditional model-heavy definition, models are specific {\em products} that can be applied to multiple domains. In the CBR model-lite definition, a model is a {\em process} that generates many products, each of which is customized to the particulars of a local domain. In this paper and ~\cite{me01c} we have seen the following advantages of CBR: easy implementation, fast runtimes, easy maintenance, able to be applied to more data, and out-performance of model-heavy methods. If these advantages apply in other problem domains, we speculate that the future of PROMISE will be ``models-as-process'' and not ``models-as-products''.


%\textbf{RQ1: Can the recommendations produced by a case based reasoning system such as {\W} perform better than a model-based approach?}\\
%
%We've demonstrated that {\W} can perform just as well if not better than SEESAW ~\ref{fig:win-loss-tie}. More importantly, similar results have been achieved while making fewer assumptions, and without the need for any model calibration.\\
%
%%\textbf{RQ2: Do different queries effect the relative performance of one method over another?}\\
%%
%%There are very few examples where The results for the OSP2 dataset[~\ref{fig:score-medians},~\ref{fig:score-spreads},~\ref{fig:effort-medians},~\ref{fig:effort-spreads},~\ref{fig:defect-medians},~\ref{fig:defect-spreads},~\ref{fig:month-medians},~\ref{fig:month-spreads}] show that there exists a discrepancy between performance in datasets with few recommendation options. 
%%
%\textbf{RQ2: In which situations should one prefer a particular method over another?}
%
%In the general case, this is an open question.
%One of the ongoing debates in effort estimation is parametric models vs case-based 
%reasoning~\cite{me10b,shepperd07}. 
%Effort estimation is a different task to quality optimization
%so it would be interpret to the results of this paper as a critique
%of parametric modeling in general.
%
%Nevertheless, there are clear indications of when case-based reasoning is {\em not}
%appropriate.
%One consideration is how much local data is available. In situations
%where, say, a company has recently formed or is breaking new ground
%in software engineering practices, the lack of relevant historical
%cases prevents the use of case-based reasoning. As Martin Shepherd says, "the use
%of accurate, systematic, historical data for building useful effort
%prediction systems is extremely important, yet in practice, such
%data is seldom available." ~\cite{01sparse} However, previous work
%has shown that one may not need much data in order to successfully
%perform CBR tasks ~\cite{me06f}. After all, "history repeats itself,
%but not exactly" ~\cite{shepperd07}.
%
%Another consideration is how satisfying a justification is from
%concrete, documented cases compared to abstract mathematical
%extrapolations? In our experience, users, especially business users,
%have found more solace in being able to use their own intuition to
%justify a recommendation. The core algorithms inside {\W} such as
%contrast sets are fast and easy to explain, bearing one accepts the
%underlying locality assumption.
%
%%XXX not sure what you are saying here. elaborate of drop
%%Finally, one must consider the lack of a single, unifying model for
%%software effort estimation. Building a model requires assumptions
%%from the underlying statistical distributions of the data to the
%%calibration parameters needed to ensure accuracy. While a model
%%benefits from reuse once created, as you only need to build it once,
%%the potential range of these parameters~\cite{me09e} can undermine
%%the usefulness of the tool as a whole.
%%
%%\subsection{Threats to Validity}
%%
%%Currently, we do not have results for datasets with actual historical measurements for defects and months. Synthesizing the defects and months for the NASA93 dataset using COQUALMO and COCOMO may have removed underlying patterns in the data. Because these values were generated from COCOMO scale factors and effort multipliers, a direct correlation exists between defects and months in regards to the project features.
%%
%%Secondly, the comparison between SEESAW and {\W} took place inside {\W}'s testing procedure. While this decision allowed a direct comparison between recommendations, the estimations generated for each were just that, estimations. We do not have a means to compare how these recommendations might perform in a real-world setting at this time.
%%
%\section{Conclusion}
%We haven't faulted previous work ~\cite{me09i}~\cite{me07g} with
%model-based approaches, but we have demonstrated that tools like
%{\W} can compete and exceed more complex methods. With only a few
%extensions to the premise of case based reasoning, {\W} has shown
%promise.
%
%
%{\W} has shown tangible potential in terms of direct performance, but the question of choosing model-based reasoning over case-based reasoning remains an ongoing discussion. Research scientists relish the idea that we can reduce complex, human-centered domains to beautiful, intuitive patterns. However, while the models we produce may be useful, effort estimation has yet to find a single, unifying abstraction under which to explain all software projects. As long as this goal remains unrealized, tradeoffs will exist between model-based reasoning and case-based.
%
%In terms of cognitive theory, CBR is a challenge to notions of science as a process of model-building.
%The mantra of that field is ``don't
%think, remember''. That is, when faced with some new situation, do
%not reason it out using some underlying model (e.g. Newton's equations
%or Boehm's parametric models).  Rather, respond to a new situation
%via an
%on-demand survey of past
%experiences \cite{schank83}. 
%Hence, CBR is also a challenge to the PROMISE conference series.
%Currently, PROMISE bills itself as ``Predictive Models in Software Engineering''. Implicit in that
%title is the assumption that model building is the right way to analyze software engineering.
%However, if CBR does as well or better than model-based methods, then we would need to rethink
%the premise of PROMISE.
%
%
