data is often noisy (i.e. contains numerous spurious signals not connected to the target
concept). Prior to experimentation, it can be hard to know what kind of learner
works best for a particular data set. 
However, as shown below, the performance of our learners on the SILAP data was
exceptionally good: in one case- nearly perfectly precise over all ten cross-validations.

\subsection{Feature Subset Selection}

SILAP's current weights were generated via the delphi sesssions 
conducted in 2005. One of the goals of this
project was to verify those weights using the historical records
in PITS. As we shall see below, some of those weightings need to
be updated.  For example, the data mining experiments described in
the next show that the factors marked with a ``\ding{52}'' contribute
most to error potential.

A
repeated result in the data mining community is
that simpler models with equivalent or higher performance
can be built via {\em feature subset selection}
algorithms that intelligently prune useless features~\cite{hall03}.
Features may be pruned for several reasons: 
\bi
\item
They may be noisy; i.e. contain
spurious signals unrelated to the target class; 
\item
They may be uninformative;
e.g. contain mostly one value, or no repeating values;
\item
They may be correlated to other variables- in which case, they can be pruned
since their signal is also present in other variables.
\ei
\noindent
The reduced feature set has many advantages:
\bi
\item 
Miller has shown that models
generally containing fewer variables
have less variance in their outputs~\cite{miller02}.
\item
The smaller the model, the fewer are the demands on interfaces 
(sensors and actuators) to the
external environment.
Hence, systems designed around 
small models
are easier to use (less to do)
and cheaper to build.
\item
In terms of this article, the most important aspect
of learning from a reduced features set is that it produces
smaller models. Such smaller models are easier to explain (or audit).
\ei

One such feature subset selector is
Kohavi \& Johns' WRAPPER algorithm~\cite{kohavi97}.
Starting with the empty
set, the WRAPPER adds some combinations of
features and asks some target
learner 
to build an effort model using just those
columns.  The WRAPPER then grows the set of
selected variables and checks if a
better model comes from learning over the larger set of variables.
The WRAPPER stops when there are no more variables to
select, or there has been no significant improvement in the learned
model for the last five additions (in which case, those last five
additions are deleted). Technically speaking, this is a hill-climbing forward select
search with a ``stale'' parameter set to 5.
For data sets as small as \fig{data}, WRAPPER terminates in a under a minute
(but for large data sets, other feature selectors would be required- see~\cite{hall03}
for a survey).

This study tried 
the WRAPPER since experiments by other researchers
strongly suggest that it is superior to many other variable pruning
methods.  For example, Hall and Holmes~\cite{hall03} compare the
WRAPPER to several other variable pruning methods including principal
component analysis (PCA- a widely used technique).  Feature
selection
methods can be grouped according to: \bi
\item
Whether or not they make special use of the
target variable in the data set such as ``runaway'';
\item
Whether
or not pruning uses the target learner.
\ei
PCA
is unique since it {\em does not} make
special use of the target variable. WRAPPER is also unique, but for
different reasons: unlike other pruning methods, it {\em does} use the
target learner as part of its analysis. Hall and Holmes found that
PCA was one of the worst performing methods (perhaps because it
ignored the target variable) while WRAPPER was the best (since it can
exploit its special knowledge of the target learner).

%
%\begin{figure*}[!t]
%\begin{center}
%\includegraphics[width=6in]{silap2.pdf}
%\end{center}
%\caption{Distributions seen ``silapClassifiedBinary''.}\label{fig:silap2}
%\end{figure*}
%\begin{figure}
%\footnotesize
%\begin{tabular}{r|@{~}r@{~}r@{\%~}l}
%$X$&\multicolumn{3}{l}{Maximum correlation of $X$ to any other factor}\\\hline
%ra &&31&\rule{15mm}{2mm}\\
%ss&&33&\rule{16mm}{2mm}\\
%do&&37&\rule{18mm}{2mm}	\\
%rm&&37	&\rule{18mm}{2mm}\\
%dt&&44	&\rule{22mm}{2mm}\\
%cx&&44	&\rule{22mm}{2mm}\\
%hs&&46	&\rule{23mm}{2mm}\\
%di&&46	&\rule{23mm}{2mm}\\
%as&&48	&\rule{24mm}{2mm}\\
%pf&&48	&\rule{24mm}{2mm}\\
%uc&&51	&\rule{25mm}{2mm}\\
%am&&53	&\rule{26mm}{2mm}\\
%fr&\rule{33mm}{2mm}&-66	&\\
%cl&&67&\rule{33mm}{2mm}	\\
%score&&68	&\rule{34mm}{2mm}\\
%ex&&73	&\rule{36mm}{2mm}\\
%us&&73	&\rule{36mm}{2mm}\\
%\end{tabular}
%\caption{Maximum correlations seen in \protect\fig{corr}.}
% \label{fig:corrs}
%\end{figure}
%
%Experimentation in software engineering is necessary. Common
%wisdom, intuition, speculation, and proofs of concepts are
%not reliable sources of credible knowledge~\cite{basili99}.
%
%Experimentation requires some control of the effect being
%studied. Such control was achieved by Basili et.al. at the
%NASA Software Engineering Laboratory (SEL)~\cite{basili02}.
%
%Basili's prefereed That work organized its experiments around four categories,
%each corresponding to different phases of the experimentation
%1) definition, 2) planning, 3) operation, and 4) intepretation.
%For example, the {\em definition} stage, resaerchs should
%\bi
%\item idenfity puprose of study 
%e.g. characterization
%evaulaton, prdiction, motivation;
%\item classify the objects of study; e.g product, process,
%model, metric or theory;
%\item
%Determine the scope of the project: e.g. single project,
%multiproject, replaicated project.
%\ei
%
%[m2]
%Measurements must be taken on both the soft- 
%ware processes and the various software products IEEE TRANSACTIONS ON SOFTWARE ENGINEERlNG, VOL. 14, NO. 6, JUNE 1988 
%Improving a product requires understanding both the 
%product and its construction processes. (M10) 
%
%[m5] Subjective as well as objective metrics are re- 
%quired. Many process, product and environment aspects 
%can be characterized by objective metrics (e.g., product 
%complexity, number of defects or effort related to pro- 
%cesses). Other aspects cannot be characterized objectively 
%yet (e.g., experience of personnel, type of application, 
%understandability of processes and products); but they can 
%at least be categorized on a quantitative (nominal) scale 
%to a reasonable degree of accuracy [4], 151, 1161, [48], 
%
%(M6) Most aspects of software processes and products 
%are too complicated to be captured by a single metric. For 
%both definition and interpretation purposes, a set of met- 
%rics (a metric vector) that frame the purpose for measure- 
%ment needs to be defined [9].
%
%(M7) The development and maintenance environments 
%must be prepared for measurement and analysis. Planning 
%is required and needs to be carefully integrated into the 
%overall software engineering process model. This plan- 
%ning process must take into account the experimental de- 
%sign appropriate for the situation [3], [14], [19], [22], 
%[66l. 
%
%(M9) The measurement process must be top-down 
%rather than bottom-up in order to define a set of opera- 
%tional goals, specify the appropriate metrics, permit valid 
%contextual interpretation and analysis, and provide feed- 
%back for tailorability and tractability [3], [ 161, [ 191, [25]. 
%
%The TAME (Tailoring A Measurement Environment) assumes that
%the measure enviornment is controllable.
%
%Even the GQM (goal question metric) paradigm assumes
%that metric collection can be controlled by the organization.
%``Goals are refined into 
%a set of quantifiable questions that specify metrics.''.
%While this is true in some cases (e.g. Basili et.al.
%describe GQM deployments at NASA, IBM, AT\&T, Burroughs,
%and Motorola~\cite[p761]{basili88,fenton99}).
%it may not be true in others. 
%
%Much metrics work focuses on data collection and analyis
%methods and less on what specific lessons can be 
%applied to the next project (basili88, fenlearned as a areu
%A recent change in NASA's funding policy for IV\&V 
%has led to much debate about appropriate software 
%processes and tools.  Previously, IV\&V had to market 
%itself to individual projects. In that model, IV\&V only 
%occurred on projects where funds could be reallocated 
%from other development activities. 
%
%That funding model has now changed. NASA headquarters now maintains
%a funding pool for IV\&V that is separate to project funds. The
%existence of this funding pool means that managers of NASA's software
%projects no longer have to make hard decisions about cutting funds
%from one part of a project in order to fund IV\&V. Instead, NASA
%projects that need IV\&V can draw on additional funding from the
%funding pool at headquarters. However, the new model has its
%administrative challenges. Before, IV\&V requirements and methods
%could be debated project- by-project amongst individual managers.
%Now, IV\&V's methods are audited at a national level.
%

%\PARstart{I}{n} the sequel, data mining and visualization methods
%will be applied to find predictors for projects that need more
%IV\&V, as well as what IV\&V tasks are most cost-effective.  In
%order to properly assess those results, we must first build an
%expectation of what we might find. Using a literature review, this
%section generates a list of hypotheses relating to software defect
%reduction.
%
%In March 2007,
%literature was gathered from three sources:
%\bi
%\item
% the NASA Software Assurance Research Program (SARP) web site\footnote{\url{http://sarpresults.ivv.nasa.gov}}.
%A search of SARP yielded on recent return-on-investment study
%by Dabney et.al., plus some forward references to papers published
%by Dabney outside of SARP~\cite{dabney06,dabney07}.
%\item
%An email survery of some researchers in the field;
%\item
%the IEEE Explore database\footnote{\url{http://ieeexplore.ieee.org/}} 
%with the query:
%\ei
%\begin{alltt}\scriptsize
%( (software<in>metadata) <and> ( independent<in>metadata ) <and> 
%( verification<in>metadata ) )<and> ( validation<in>metadata )
%\end{alltt}
%
%The
%IEEE Explore 
%The results ranged mostly from 1988 to 2007 (with one
%outlier from 1981) and included
%two revisions to the IEEE 1012 standard on software 
%verfication~\cite{1012} and the releated standard on distributed
%ineractive simulation~\cite{1278.4}.
%Some hardware-only papers were found and these 
%were deleted. The resulting corpus contained 72 papers
%and the standard text on IV\&V~\cite{lewis90}. 
%
%Much of the corpus took the route of 
%(e.g.) Krauskopf \& Rash~\cite{krauskopf90} who argued in
%general terms for the value of IV\&V. Like Lewis~\cite{lewis90},
%Krauskopf argued for for lifecycle artifacts to be complete, logical,
%consistent, testable, understandable, and 
%have quanitifable and measurable performances goals. However, this part
%of the corpus made no offers regarding the most cost-effective method of
%assessing these artifacts. 
%
%The corpus was then studied for detailed
%claims with some measure of support
%and some generality beyond their immediate applciation area.
%Several stand out publications were seen, listed here
%in date of publication:

