
\section{Conclusion}
\noindent {\em To attain knowledge, add things every day.\newline
To attain wisdom, remove things every day.\newline
                        -- Lao-tse}

Researchers into empirical methods such as Cohen~\cite{cohen95}
advise
benchmarking
supposedly sophisticated method against 
simpler alternatives.
At the very least, simpler methods may offer a baseline result that
demonstrates the value of the sophisticated approach.
For learning changes to software projects,
{\W} is a candidate baseline tool since:
\bi
\item
It is simple to implement and fast
to execute; 
\item
The algorithm scales well (runs in log-linear time);
\item
For the data studied here, it performs  as least as good (or better) than a range of model-based
methods.
\ei
While an interesting baseline result, this paper is hardly a final result.
Our hope is that this paper inspires more research into instance-based planning algorithms
that learn how to change some context in order to achieve/avoid certain kinds of results.
There is much to explore. For example, there are many open issues in the above discussion:
\bi
\item Are model-based methods worse for noisier data?
\item Is ``data cleansing'' recommended for regression, but deprecated for  instance-based methods?
\item How best to reduce spread, thus increasing the confidence a user has in the results?
\ei
As to this last point, better algorithms may not be effective in reducing spread.
Elsewhere we have tried reducing estimate variance with feature
subset selection (FSS)~\cite{chen05,me06d}.  Sadly FSS only somewhat
reduces, but does not tame, the variance problem.
Nor should we look to more extensive data collection (recall the discussion in \tion{draw}
on the difficulties with getting process data).
Jorgensen~\cite{jorg04} advises that the best estimates come from combining results from multiple methods,
including human-based judgment. Accordingly,
in our own work, we have started exploring Bayesian frameworks that
combine the output of instance-based methods with that of human experts
(but
have no results to report at this time).



%Software Engineering is based on the mathematical idea that computation can be fully specified. However, software remains an essentially human process, wrought with social and creative underpinnings.  Unlike traditional engineering concerns like mechanical forces or material properties, groups of humans are (currently) much less consistent and predictable.
%
%Yet, we still seek general truisms about software just as we would a physical phenomenon. For example, models like COCOMO abstract project qualities into quantified effects on development effort. These model-based approaches can combine both expert advice and local data in order to generalize complex, human behavior to a handful of variables. However, models must be calibrated to specific institutions and respond slowly to changes in development techniques. 
%
%One may question whether it is necessary to construct the model in the first place. A project manager's concern isn't beauty in generalization, but a practical one: {\em how to find useful changes for a particular project}. The drawbacks associated with model uncertainty and relevancy offer room for alternatives to model-based methods.
%
%Instance-based methods eschew generalizations about a constantly shifting field. Instead, inferences are made from local data alone. While model-based approaches can operate without any historical data, model calibration and validation do require historical instances. If you already have the data, you might as well use it.

%{\W}2 offers a simple, fast implementation of instance-based reasoning using contrast sets. Improvements to software projects are gleaned by answering a simple question: {\em Given a set of similar projects, which attributes describe the most desirable projects but not the least desirable projects?} We set out to compare this simple learner with a related model-based algorithm called SEESAW.
%
%\fig{cocomo81results} show our results. {\W}2 delivers statistically better recommendations across multiple runs for 11 of 24 trials. More importantly, when {\W} did not statistically outperform SEESAW, it still performed no worse than SEESAW.
%
%{\W}2's instance-based approach is also data agnostic. Whereas
%SEESAW depends on data expressed in COCOMO format, we are able to
%run {\W}2 on several arbitrary effort estimation datasets
%(see~\fig{other-reduce} and ~\fig{data}). In these cases, {\W} always
%reduces median software effort and spread uncertainty by at least
%25\%, with some datasets seeing reductions of 75\% or more.
%
%From these results we conclude that {\W}2 offers better or comparable performance to a model-based method while remaining simpler and more flexible. With as few as 15 data points (see Kemerer dataset in ~\fig{data}) {\W} is able to suggest recommendations that consistently estimate effort reductions by over 50\%. 
%
%
%We conclude that:
%\be
%\item
%Often, managers have many options for effectively changing and
%improving their projects.
%\item
%Very simple instance-based methods can  do better than more
%complex model-based methods.
%\ee
%While the first point  may be of most interest to
%industrial practitioners, it is the second point  that may be most interesting to researchers.
%There are any number of sophisticated methods for
%exploring the complexities and uncertainties associated with
%controlling software engineering projects\footnote{
%E.g. 
%Bayes nets;
%fuzzy logic;
%constraint-satisfaction
%simulated annealing; evolutionary algorithms;
%tabu search; ant-based search; various data mining tools such as
%support vector machines or random forests;
%numeric methods such as integer programming; 
%logic-based-approaches such as Davis-Putnam procedures or binary-decision diagrams; 
%or 
%any number of AI search algorithms~\cite{lessmann09,norvig03,Gu97,me09g,uribe94,harman07}.}. 
%Nevertheless, before researchers explore sophisticated methods, they should
%try a simpler
%method, at least for the purposes of establishing a performance baseline.
%For learning changes to software projects,
%{\W} is a candidate baseline tool since:
%\bi
%\item
%It is simple to implement and fast
%to execute; 
%\item
%The algorithm scales well (runs in log-linear time);
%\item
%For the data studied here, it performs  as least as good (or better) than a range of model-based
%methods.
%\ei
%
%

%We've demonstrated that our simple, fast {\W}2 algorithm performs as well or better than a selection of complex, model-based methods using COCOMO. We've also shown it possible that {\W}2 can be instantly applied to arbitrary datasets without requiring calibration ~\ref{fig:other-reduce}. {\W}2 exchanges generalities for fast, local recommendations. However, to do so requires a shift from typical model-based approaches. For, say, organizations without a historical case base, it is not possible to recommend {\W}2.

%Nevertheless, other areas of research advise that
%researchers should at least baseline their supposedly more sophisticated method against a
%simpler alternative~\cite{cohen95,holte93}. Accordingly, 
%we offer  {\W}2 as a baseline device for learning changes to
%software projects since.

%However, instance-based methods represent a way to better certify the relevance of a recommendation. Where models tend to be generated once and rarely updated, {\W}2 offers a way to instantly conform to new software engineering techniques: simply add more cases. Therefore, the crux of recommending {\W} lies on the task at hand. For reasoning about large, static systems with an existing, well-tuned model, the burden of model generation isn't as important. However, software estimation represents a rapidly moving target, and it is difficult to claim that a single model will continue to represent rapid shifts in development environments.

%In the end it is all about the data. If you have it, use it. While models can be adapted, doing so may not be necessary. Instead, we have shown success in exchanging model-based generalities for instance-based local lessons. While we cast out grand, general statements on effects in software engineering, we must ask ourselves whether we even {\em need} them. Software estimations are like a hitting a moving target. Instead of trying to deduce its exact location, perhaps we should instead look at our previous shots and adjust accordingly.



