\comment{
\cite{shepperd97} investigated case based reasoning with
k nearest neighbor make effort estimations.
Having reported successful results a follow-up
paper \cite{kirsopp02search} was written that describes how 
to increase the quality of CBR estimation by 
doing feature subset selection.
This paper compared using random search, hill climbing,
and forward sequential selection.
The comparision metric was the absolute residual error
from a jack-knife validation over the 407 records.
Another popular metric is $\chi^2$, but this
was rejected by the authors because they felt
it was too risk-adverse 
\footnote{This is due to the mathmatical properties
of the square function.  The square function
is concave up, which means that small numbers
of large errors can dominate a $\chi^2$ function}.
4028 subsets were generated randomly, with the average
residual error being insignificantly larger than
the residual from using all features.  The best
set found, however, resulted in a 35\% reduction
in residual error.  This suggests that while
a random subset is likely to perform at the same
level as the entire feature set, there do exist
subsets that will outperform the entire set.
In the second experiment a hill climber (with multiple
restarts) was used.  The neighborhood examined
for each candiate solution was all solutions with
only one feature either removed or added.
113 restarts were tried, with the mean
residual error being 46\% lower than the residual
when using all features.
The authors then present a scatter plot showing
the relationship between the number of features showed
and the residual error.
The plot shows a clear coorelation between using more
features and increasing residual errors.
This motivated the authors to try a forward select
algorithm.
A forward select can be thought of as a steepest-ascent
hill climber, but with the inital solution using no
features and features never being removed from the candiate
solution.  A backward selection (where the initial solution
uses all the features) was not considered because of the coorelation
discovered from the scatter plot.
The result of the foward select was insignificantly worse
than the best result from the hill climber restarts,
but required 1/3 the number of evaluations of the objective
function.
After presenting the results of the three different feature subset
selection algorithms, the authors discuss the objective landscape.
They conclude several features of the landscape.
It is very multi-modal.  The hill climbers found 98
distinct peaks (out of 113 restarts), 90 of which
outperformed the best set found during the random search.
It is very continuous, the median number of moves for
the hill climber was 15 (with a max of 43).
Visualization of the different peaks found revealed
an overall trend in the location of the best peaks
found.  But a restricted random search in the area
of the best peaks still performed poorly.
This suggests the peaks are very steep, although
located on top a smoothly increasing landscape.
Finally since all of the twice visited peaks were in the
top ten, these high peaks likely have a large basin of attraction.
}
