
\begin{figure}
{\footnotesize
\begin{tabular}{|r@{~=~}p{3.2cm}|r@{~=}p{3.2cm}|}\hline
\multicolumn{2}{|c|}{Derived}&
\multicolumn{2}{|c|}{Raw features}\\\hline
	co&Consequence&

am&Artifact Maturity\\

	dv&Development&


as&Asset Safety\\

	ep&Error Potential& 

cl&CMM Level\\
	pr&Process&
cx&Complexity\\
	sc&Software Characteristic& 
di&Degree of Innovation\\

\multicolumn{2}{|c|}{~}&do&Development Organization\\
\multicolumn{2}{|c|}{~}&dt&Use of Defect\newline  Tracking System\\
\multicolumn{2}{|c|}{~}&	ex&Experience \\
\multicolumn{2}{|c|}{~}&	fr&Use of Formal Reviews\\
\multicolumn{2}{|c|}{~}&	hs&Human Safety\\
\multicolumn{2}{|c|}{~}&	pf&Performance\\
\multicolumn{2}{|c|}{~}&	ra&Re-use Approach\\
\multicolumn{2}{|c|}{~}&	rm&Use of Risk\newline Management System\\
\multicolumn{2}{|c|}{~}&	ss&Size of  System\\
\multicolumn{2}{|c|}{~}&	uc&Use of CM\\
\multicolumn{2}{|c|}{~}&	us&Use of Standards\\\hline
\end{tabular} }
\caption{The SILAP features. Project idenitifies are shown top
left. Derived features are shown bottom left- and these
are derived from the 16 raw features shown on the right hand side.}\label{fig:silapFeatures}
\end{figure}
\begin{figure}[!t]
{\scriptsize
\begin{alltt}
function CO( tmp) { 
        tmp=0.35*AS + 0.65 *PF; 
        return (round((HS) > tmp) ? HS : tmp) } 
function EP() { 
        return round(0.579*DV() + 0.249*PR() + 0.172*SC()) } 
function SC() { 
        return 0.547*CX + 0.351*DI + 0.102*SS } 
function DV() { 
        return 0.828*EX + 0.172*DO } 
function PR() { 
        return 0.226*RA + 0.242*AM + formality() } 
function formality() { 
        return 0.0955*US + 0.0962*UC + 0.0764*CL  +
               0.1119*FR + 0.0873*DT + 0.0647*RM }
\end{alltt}}
\caption{silap.awk: computes the derived features from the raw features of \protect\fig{silapFeatures}.}\label{fig:silap.awk}
\end{figure}
\begin{figure}
\begin{center}
\footnotesize
\begin{tabular}{p{1.2cm}|rrrr|r|r@{}r}
& & & & weight*& contribution&\multicolumn{2}{|c}{fss?}\\
goal & feature & weight & filter & filter & to goal&5&4\\\hline
    consequence&      hs&   0.35&      1&0.350& 35\%&&\\
     (co)&      pf&   0.65&      1&0.650& 65\%&&\\\hline
     error&      ex&  0.828&  0.579&0.479& 47\%&\ding{51}&\ding{51}\\
     potential&      cx&  0.547&  0.172&0.094&  9\%&&\\
     (ep)&      do&  0.172&  0.579&0.100&  9\%&&\\
     &      di&  0.351&  0.172&0.060&  6\%&&\\
     &      am&  0.242&  0.249&0.060&  6\%&\ding{51}&\ding{51}\\
     &      ra&  0.226&  0.249&0.056&  5\%&&\\
     &      us& 0.0955&  0.249&0.024&  2\%&\ding{51}&\ding{51}\\
     &      uc& 0.0962&  0.249&0.024&  2\%&&\\
     &      fr&  0.119&  0.249&0.030&  2\%&&\\
     &      dt& 0.0873&  0.249&0.022&  2\%&&\\
     &      ss&  0.102&  0.172&0.018&  1\%&&\\
     &      cl& 0.0764&  0.249&0.019&  1\%&&\\
     &      rm& 0.0647&  0.249&0.016&  1\%&\ding{51}&
\end{tabular}
\end{center}
\caption{Relative contribution of \protect\fig{silap.awk} factors to the goals of consequence and error potential.
The percentages shown on the right are normalized values for the $weight*filter$ column. }\label{fig:cons}
\end{figure}

\subsection{SILAP}
The SILAP model
({\em Software Integrity
Level Assessment Process}) was built via
extensive 
delphi sessions amongst NASA civil servants.
Predictors were written for error potential (a.k.a. liklihood) and conseqeunce of error (a.k.a. risk).
These predictors held
the group's consensus belif on effects the quality of NASA projects.
\fig{silapFeatures} shows the
SLAP ``derived'' factors that are computed from the ``raw'' factors using the functions of \fig{silap.awk}.
These computed error potentials and consequences were then used to select IV\&V tasks, using
\fig{silapModel}.


 SILAP grew into an early lifecycle project assessment methodology for IV\&V.
In operation, it takes two weeks (or more) full-time work to do determine
values for the raw features of \fig{silapFeatures}.
In that two week period, IV\&V analysts
review project documents;
offer a score for each SILAP feature;
and write a detailed rationale for that score.
Most of  the SILAP features are scored low to high, 1 to 5 (respectively)
or ``0'' for ``don't know''.  

NASA civil servants at IV\&V
report that SILAP has simplified and clarified their discussions
with projects regarding what IV\&V tasks are/are not to be performed.
One of the advantages of SILAP is that, since it was built locally at
IV\&V, it is
understood locally.
Previous risk models, which had been developed elsewhere,
were to a degree ``black box'' models that were 
poorly understood. Hence, it was harder to defend their conclusions
to clinets who were lobbying
for less IV\&V oversite.

Each raw SILAP factor is weighted and, usually, filtered by a second numeric to derive
the consequence and error potential values.
The {\em weight*filter} column of \fig{cons} shows the relative contribution of each raw
factor, as found by the delphi sessions amongst the NASA civil servants.
The following study is the first experimental
verification of the correctness of those
relative contributions.

\subsection{Historical Records}

This study could access
SILAP descriptions of 280 sub-systems from 11 NASA projects. All these projects
received IV\&V in the period 2005 to 2007.
Of these, only   six of those  projects could be mapped into the issue tracking report
system.
Therefore this study was based on data from 211 sub-systems taken from 6 projects.

Each sub-system was described  using:
\bi
\item
the system name: $L1$;
\item
 the sub-system type: $L2$. These types include GNC (guidance and navigation);
PAYLOAD (specialized
payloads); CDH  (command \& data handling), and GROUND (ground systems).
\item
the 16 raw SILAP features of \fig{silapFeatures};
\item
and five counts $s_1,\ldots s_5$ showing
the number ofissues at differing severity levels. 
\ei
 In this data,
the frequnecy of issues, relative to issue 1 
issues, was 1:24:206:127:23 for $s_1, s_2, s_3, s_4,s_5$ issues, repsectively.
That is, level 3 issues were the majority and level 1 issues were relatively rare.

For the purposes of analysis,
the severity counts were summarized 
as follows. Let $s_{i,j}$ be the frequency of severity issue $i$ in sub-system
$j$. If $Max_i$ is the maximum number of issues of severity $i$, totalled
across all sub-systems, 
then $W_j$ is a weighted sum
where all severity one issues are multiplied by  $10^5$, severity two issues
are multipled by $10^4$, and so on:
\begin{equation}\label{eq:w}
W_j= 1 + \sum_{i=1}^5\left( \frac{s_{i,j}}{1+Max_i}{\cdot}10^{6-i}\right)
\end{equation}
Note that:
\bi
\item
The severity counts were divided by the maximum severity counts so that any
severity $i+1$ error out-weighed any severity $i$ error. This models the business
intuition that (e.g.) a single servity one issue was more important than any number
of severity two errors.
to compute
consequence (CO) and error potential (EP).
\item
The 
``plus one'' parts of this equation were added to avoid numeric errors. 
In this data, the severity frequency counts were
large enough to make the ``plus one'' contributions relatively negligable.
\ei
\begin{figure}
\begin{center}
\includegraphics[height=2.25in]{tim/coepissues3D.pdf}
\includegraphics[height=1.5in]{tim/coepissuesCOW.pdf}
\includegraphics[height=1.5in]{tim/coepissuesEPW.pdf}
\end{center}
\caption{SILAP's computed error potentials (ep), consequence (co), and weighted severities count (W) from
\protect\eq{w} seen in 211
NASA sub-systems from 6 systems. The bottom two plots are 2D projections of
the top 3D plot.}\label{fig:epcow}
\end{figure}

\begin{figure}
\definecolor{gray}{gray}{0.8} 
\definecolor{g}{gray}{0.81} 
\scriptsize
\begin{center}
\begin{tabular}{r@{~}|r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r}
&as&pf&ex&do&us&uc&cl&fr&dt&rm&ra&am&cx&di&ss&$W$\\\hline
hs&24&18&4&12&-10&34&-20&43&44&3&7&13&8&46&21&-21\\
as &&48&-20&-7&-25&-4&-30&16&24&-10&0&-7&42&23&9&-25\\
pf & &&-9&-16&-21&-5&-26&5&19&1&13&-2&44&11&12&-11\\
ex & & &&-12&\colorbox{gray}{73}&29&\colorbox{gray}{67}&-27&-3&-28&-3&30&-29&33&29&\colorbox{gray}{68}\\
do & & & &&6&9&4&30&-20&37&-23&4&-27&2&-23&-3\\
us & & & & &&37&\colorbox{gray}{58}&-25&-4&-21&-22&11&-43&24&7&\colorbox{gray}{56}\\
uc & &  & & & &&37&\colorbox{gray}{51}&41&13&-3&47&3&30&7&-18\\
cl & & & & & & &&-26&-28&-7&-6&\colorbox{gray}{53}&-25&10&23&\colorbox{gray}{65}\\
fr & & & & & & & &&35&32&31&0&20&12&-21&\colorbox{gray}{-66}\\
dt & & & & & & & & &&2&-8&-14&35&31&23&-37\\
rm & & & & & & & & & &&28&23&2&-20&-30&-18\\
ra & & & & & & & & & & &&14&28&-7&11&-1\\
am & & & & & & & & & & & &&4&12&14&17\\
cx &  & & & & & & & & & & & &&4&27&-33\\
di & & & & & & & & & & & & & &&31&13\\
ss & & & & & & & & & & & & & & &&33\\
\end{tabular}
\end{center}
\caption{Percent 
correlations within 211 sub-systems
between SILAP parameters scored 1,2,3,4,5 and the $W$ score from \protect\eq{w}.
The nine pairs with \mbox{$correlation \le -50\%$ or  $correlation \ge 50\%$} are highlighted
in a \colorbox{gray}{gray box}.
}\label{fig:corr}
\end{figure}


\fig{epcow} shows the relationship between the outputs derived
by SILAP (from \fig{silap.awk}) and the weighted sum $W$ of the servity reports.
If SILAP was an accurate predictor for problems, we would expect some correlation between
$W$ and {\em either} 
consequence of error (co) {\em or} error potential (ep). A visual inspection suggests that
neither correlation exists.

\fig{corr} raises other doubts about the efficacy of the current version of SILAP.
Correlation\% varies from -100\% to 100\% and a correation of zero means that there is no numeric
correspondance between two variables.
As shown in \fig{corr},
most of the correlations in our data have correaltions close to zero.
In terms of building succinct models, this is a useful result since it suggests that
a predictor for issue severity might be built using just a small subset of the SILAP features.
We will return to this point, below.

In only nine cases (those shown with gray boxes in \fig{corr}) 
are the correlations above 50\% or below -50\%.
Some of those correlations corresponds to established wisdom in the field. For example,
Basili \& Shull \& Lanubile~\cite{basili99}
would accept the strong correlation between increasing
the use of formal reviews (fr) and decreasing high severity issues (correlation=-66\%).
However, we know of no researcher who would accept the following abherent results:
\bi
\item
Increasing programmer experience (ex) 
{\em increases} high severity scores ($W$), correlation=68\%.
\item
Increasing CMM-level (cl) {\em increases} high servity scores ($W$), correlation=65\%.
\ei

A review of our data  explains these abherent results. Recall that
\eq{w} weighted severity one errors by a factor of 100,000. Hence,
even a small system with just a sfew sub-systems
can have a disproportinate impact on the correlations
if that system has severity 1 issues.
This is indeed the case in our data.
One system, with relatively few sub-systems had most of the
severity one issues. The managers of that system
were highly concerned about the
quality of their product. Hence, they employed numerous best practices
including high CMM practices (cl), extensive use of standards (us),
hiring very experienced developers (ex). That is, these practices did not
{\em cause} the severity one issues in that system. Rather they were adopted
{\em in response} to the presence of those issues. 

 
Based on the above, we  comment that:

\bi
\item
\fig{epcow} shows that
SILAP's calcuated consequence and error potential values are poor predictors for
issue severity.
Hence, the current version of SILAP needs revision. 
\item
\fig{corr}
shows that only a subset of the current SILAP features are connected to other features.
Hence, 
a predictor for issue severity might use only a subset of the
current SILAP features. 
\item
The percentage of level 1 severities in our sample is so small that quirks in
a single system can lead to abherent results. Hence, it is unlikely that we
will be able to build effective predictors for level 1 severity issues. Accordingly,
in the sequel
we will experiment with combining level 1 and level 2 issues into a single ``worst case''
target class.
\ei


\section{Learning a new Version of SILAP}

\PARstart{T}{wo} data mining methods were employed to learn a new version 
of SILAP:
\bi
\item
The RIPPER~\cite{cohen95r} rule learner was applied to generate
a set of rules that predict for issue severity.
\item
RIPPER was ``wrapped'' in a feature selector that experimented with subsets of the
available SILAP factors. 
\ei

Both methods are standard techniques for generating succinct, easily explainable,
models from data.
Other data mining methods were not employed since they are known to generate
superflous reduncancies.
For example, tree learners such as C4.5~\cite{quinlan92}
or CART~\cite{breiman84}
execute in local top-down
search, with no memory between different branches.  Hence, the same concept
can be needlessly repeated many times within the tree.
Consequently, such trees can be  cumbersome, needlessly large, and
difficult to understand. 

RIPPER, on the other hand,
takes an aggressive approach to pruning away superflous parts of a model:
\bi
\item 
After building a {\em rule}, RIPPER performs a back-select to see what rule
conditions
can be deleted, without degrading the performance of the rule.
\item
Similarly,
after building a {\em set of rules}, RIPPER performs a back-select to see what {\em rules}
can be deleted, without degrading the performance of the rule set.
The learned rules are built while minimizing  their {\em description length}.
This is an information theoretic measure computed from the size of the learned rules,
as well as its errors. If a rule set is over-fitted, the error rate increases,
the description length grows, and RIPPER applies  rule set pruning.
\item
RIPPER also employs a novel
{\em rule set optimization} algorithm that tries
replacing rules
some straw-man alternatives (i.e. rules grown very quickly).
\ei

RIPPER is one of the fastest rule leaners currently known in the literature. In its
haste, it may miss parts of the model that can be pruned. Hence, a much slower
and more thorough feature pruner was also employed:
\bi
\item 
Kohavi \& Johns' WRAPPER algorithm~\cite{kohavi97}
explores subsets of the available features. Each subset is assessed by
asking some target
learner (in our case, RIPPER)
to build a model using just that subset.
\item
The WRAPPER grows the feature
set while the target learner reports that the larger set does no better than the subset.
\item
WRAPPER stops when there are no more variables to
select, or there has been no significant improvement in the learned
model for the last five additions (in which case, those last five
additions are deleted). Technically, this is a hill-climbing forward select
search with a ``stale'' parameter set to 5.
\item
In empirical comparisons with other feature pruners, Hall \& Holmes 
report that WRAPPER is  slow, but yields the best results~\cite{hall03}.
\ei

\subsection{Pre-Processing}

RIPPER's rules predict for discrete classes so, prior to learning, the \eq{w} results
for the 211 sub-systems were sorted and divided into {\em bins} with equal population size
(equal frequency discretization~\cite{dou95}). The \fig{corr} results
showed some abherent results resulting from a small number of projects with level 1 severities.
Accordingly, the learning experiments were conducted twice:
\bi
\item
Once where the \eq{w} values were divided into five target classes: \_1, \_2, \_3, \_4, \_5
representing the bins containing (left to right) the most to least high severity issues.
\item
Once where the \eq{w} values were divided into four target classes, generated as follows.
After applying five-binning, the two bins representng severity 1 and severity 2 issues
were  combined  into one. In the four class system, we denote 
the combined class as
$\_1'=\_1 \cup\_2$. 
\ei
\begin{figure}
\begin{center}
\footnotesize
\begin{tabular}{r|rr|r}
&\multicolumn{2}{c|}{number of classes}\\\cline{2-4}
feature	&4&	5	&average\\\hline
us&		10&	10&	10.0\\
am&	    9&	9&	9.0\\
ex&		8&	9&	8.5\\
rm&		6&	9&	7.5\\
ra&		5&	5&	5.0\\
ss&	    5&	4&	4.5\\
fr&		6&	1&	3.5\\
uc&		6&	1&	3.5\\
do&		3&	3&	3.0\\
cl&		4&	2&	3.0\\
dt&		2&	&	2.0\\
pf&		2&	&	2.0\\
cx&		2&	2&	2.0\\
di&		2&	1&	1.5\\
as&		2&	1&	1.5\\
hs&		1&	1&	1.0
\end{tabular}
\end{center}
\caption{Number of times a feature was selected in 10y
experiments where features were assessed using 90\% of
the data (selected at random).}\label{fig:fss}
\end{figure}

\begin{figure*}
\footnotesize
\begin{center}
\begin{tabular}{r|cr|rrrrr|rl}
&&\multicolumn{1}{c}{~}&\multicolumn{5}{c}{target classes}\\\cline{4-8}
%id&number of classes&attributes uses&good&ungood&&&&F&\\\hline
%2a&2&l2&0.80&0.26&&&&0.53&\rule{18mm}{2mm}\\
%2b&&l1&0.91&0.86&&&&0.89&\rule{30mm}{2mm}\\
%2c&&raw&0.95&0.95&&&&0.95&\rule{32mm}{2mm}\\
%2d&&fr+ex&0.97&0.95&&&&0.96&\rule{32mm}{2mm}\\
%2e&&fr+ex+hs&0.99&0.97&&&&0.98&\rule{33mm}{2mm}\\
%2f&&l1+l2&1.00&1.00&&&&1.00&\rule{33mm}{2mm}\\\hline
%\multicolumn{10}{c}{~}\\
treatment&number of classes&features&\multicolumn{2}{l}{$\_1'=\_1\cup \_2$}&\_3&\_4&\_5&F&\\\hline
4a&4&L2&0.57&&0.00&0.31&0.00&0.22&\rule{7mm}{2mm}\\
4b&&us&0.64&&0.60&0.00&0.00&0.31&\rule{10mm}{2mm}\\
4c&&L1&0.67&&0.69&0.00&0.46&0.46&\rule{15mm}{2mm}\\
%4d&&l1+l2&0.96&0.89&0.94&0.68&&0.87&\rule{30mm}{2mm}\\
4d&&all 16 raw SILAP features&0.93&&0.95&0.87&0.90&0.91&\rule{30mm}{2mm}\\
4e&&us+am+ex+uc+fr+rm&0.95&&0.94&0.97&0.92&0.94&\rule{31mm}{2mm}\\
4f&&ex+us+am&0.95&&0.96&0.955&0.94&0.95&\rule{32mm}{2mm}\\\hline
\multicolumn{10}{c}{~}\\
treatment&number of classes&features&\_1&\_2&\_3&\_4&\_5&F&\\\hline
5a&5&L2&0.00&0.46&0.00&0.31&0.00&0.15&\rule{5mm}{2mm}\\
5b&&L1&0.00&0.54&0.69&0.00&0.46&0.34&\rule{11mm}{2mm}\\
5c&&us&1.00&0.58&0.60&0.00&0.00&0.44&\rule{15mm}{2mm}\\
%5d&&l1+l2&1.00&0.94&0.86&0.94&0.72&0.89&\rule{30mm}{2mm}\\
5d&&all 16 raw SILAP features&0.97&0.94&0.93&0.92&0.94&0.94&\rule{31mm}{2mm}\\
5e&&us+ex+am+rm&0.97&0.95&0.96&0.94&0.93&0.95&\rule{32mm}{2mm}
\end{tabular}
\end{center}
\caption{$f$ measures and $F$ results for different learning treatments.
The \_1' class contains the union of all the \_1 and \_2 sub-systems. }\label{fig:ff}
\end{figure*}


\fig{fss} shows the WRAPPER results for four and five target classes.
The WRAPPER was run ten times using RIPPER as the target learner.
Each run used a randomly selected 90\% sample of the data.
As predicted by the \fig{corr} results,
best predictors were found using a small subset of the RIPPER features.
In the four class
experiment:
\bi
\item Use of standards (us) was always selected.
\item
Artifact maturity (am) and developer experience (ex)
were often selected (8 times or more). 
\item
Use of formal reviews (fr), use of configuration
managent (uc), and use of a risk management system (rm) were selected six times.  
\ei
In the five class experiment, fewer features were selected:
\bi
\item
Use of standards (us) was found in all ten runs.
\item
Artifcat maturity (am), developer experience (ex), and use of a risk management system (rm)
were selected nine times out of ten.
\ei

\subsection{Experimental Design and Evaluation}

Experiments were then conducted with RIPPER, using various feature subsets:
\bi
\item 
The subsets of selected features found in the four and five class feature pruning experiments
of \fig{fss}. 
\item
All the {\em raw} SILAP features. 
\item
Just the system name: $L1$. 
\item
Just the sub-system type: $L2$. 
\ei

The results were assessed using $F$; i.e. the  average f-measure
seen in all the target classes. 
$F$ is defined as follows.
Let $\{A_x,B_x,C_x,D_x\}$ denote the
true negatives, 
false negatives, 
false positives, and 
true positives
(respectively) for class $x$.
Various performance measures can be computed from these values. 
For example, recall (or $pd$) comments on how much of the target was found.
\begin{equation}
pd_x=recall_x=\frac{D_x}{B_x+D_x}
\end{equation}
Also, precision (or $prec$) comments on how much of the instances that
triggered the detector actually containing the target concept.
\begin{equation}
prec_x=precision_x=\frac{D_x}{D_x+C_x}
\end{equation}
The $f$-measure is the harmonic mean of precision and
recall. It has the property that if {\em either} precision or recall
is low, then the $f$-measure is decreased. The $f$-measure is useful
for dual assessments that include {\em both} precision and recall.
\begin{equation}
f_x=\frac{2 * prec_x * pd_x}{prec_x + pd_x}
\end{equation}
Given $n$ target classes {\em x,y,\ldots}, then one measure of the total
performance is $F$, i.e. the average $f$ measure seen in  all classes:
\begin{equation}
F=\frac{1}{n}\sum_i^nf_i
\end{equation} 
This measure
has the range $0\le F\le 1$. Note that  the {\em larger} the $F$ value, the {\em better}
the learned model for predicting all the target classes.

The $F$ measures were collected in
10-way  cross-validation
experiments. Such experiments assess a learned model using data {\em not}
used to generate it.
To conduct such an experiment,
the available date is divided into $N$ buckets.
For each bucket in a $N=10$-way cross-evaluation, a model is
learned on the nine of the buckets, then tested on the remaining
bucket.  
These studies
are the 
referred evaluation method when the
goal  is to produce predictors intended to  predict future 
events~\cite{witten05}.

\begin{figure}
\scriptsize
\begin{alltt}
us >= 5                         => \_1 
am >= 3                         => \_5 
ex <= 2 AND rm <= 1 AND us >= 2 => \_5 
us <= 1 AND am >= 2             => \_5 
ex >= 2 AND am <= 1 AND us <= 2 => \_5 
ex <= 1 AND us >= 2             => \_4 
us >= 3 AND ex >= 3             => \_4 
ex <= 1                         => \_3 
us <= 2 AND ex <= 2             => \_3 
otherwise                       => \_2 
\end{alltt}
\caption{Rules from treatment 5e or \protect\fig{ff}. $F=0.95$. }\label{fig:5e}
\end{figure}
\begin{figure}
\scriptsize
\begin{alltt}
am >= 3 and us >= 3                         => \_5 
ex <= 2 and am >= 2 and us <= 1             => \_5 
am <= 1 and us >= 2 and us <= 2 and ex >= 2 => \_5
ex <= 1 and us >= 3 and am <= 1             => \_5 
ex <= 1 and us >= 2                         => \_4 
us >= 3 and ex <= 4 and ex >= 3             => \_4 
us <= 2 and ex <= 2                         => \_3 
otherwise                                   => \_1' 
\end{alltt}
\caption{Rules from treatment 4f of \protect\fig{ff}.
$F=0.95$. }\label{fig:4f}
\end{figure}

\subsection{Results}

\fig{ff} shows the $F$ values seen in 
10-way
{\em cross validation} experiments using RIPPER on 
on the four and five class systems.   
Seveal aspects of  \fig{fss} are noteworthy:
\bi
\item
No single feature was a best predictor for issue severity. All the experiments
with single features (L1,L2,us) performed worse than those that used
combinations of features. This explains a curious effect- single
features with poor correlations
to $W$ such as artificat maturity (am) (see \fig{corr}) proved to be quite useful.
\item
The high-ranked features listed in \fig{fss} 
performed better than using just the raw SILAP features; i.e. the WRAPPER
experiments were a success.
\item
Using just system name ($L1$)  performed poorly. This is a good since
otherwise, 
there would be no generality in NASA's issue reports.
\item
The sub-system type ($L2$) proved to be a poor predictor of issue severity.
That is, there was no evidence from this data that any of
GNC, GROUND, CDH, or PAYLOAD are inherently more risky that any of the others.

\ei

The rules learned from \fig{fss}'s best treatments  are shown in \fig{5e} and \fig{4f}.
It is important to read these rules top to bottom since
a rule fires {\em only} if {\em all} the rules above it {\em fail}. 

The rules of \fig{5e} are 
abherent- the top rule says that use of standards (us) predicts for highest severity issues.
Recalling the above discussion, this abheration is due to quirks in the single project
with the highest severity errors. 

The rules of \fig{4f} combine the severity 1 and two issues into one class \_1'.
This avoids the problem of a small sample of severity one issues.
This rules seem saner than \fig{5e}. For example:
\bi
\item
The first rule
claims that mature products being extended using industrial standards
predicts for very low issue severity.
\item
The last rule predicts for high severity issues and that can be reached via poor use of
standards and poor developer experience.
\ei
Certain quirks with \fig{4g} can be explained with refernce to the correlation matrix
shown above.
For example, proponents of the CMM model (e.g.~\cite{gibson06})
might raise a concern that these rules make no mention of CMM as a method of lowering 
defects.  However, \fig{corr} reports a strong correlation (67\%)
between CMM level (cl) and developer experience (ex). Given such a strong correlation,
a feature pruner such as WRAPPER could select either of cl {\em or}  ex for the final model.
That is, process maturity is tacit in \fig{4g} via the correlated concept of developer
experience. 

\subsection{Discussion}

The case was made above that the current version of SILAP needs revising. While
the derived features of SILAP are clearly problematic (see \fig{epcow}), the raw SILAP
features produce remarkably good predictors for issue severity.

The results of \fig{ff} are actual better than they may appear.
Cross validation 
report the performance of a model that was learned using {\em less} than all the available data.
Hence, they
{\em under-estimate}
the performance of a learned models so the {\em actual} performance of our best learned models 
is {\em more than} the $F=0.95$ reported in \fig{ff}.
That is, data mining the SILAP data yielded near
perfect predictors for weighted issue severity
(i.e. nearly
perform at $F=1$). 

Since the rules of \fig{ff} are working so well, we recommend deploying them as a replacement
for the SILAP error potential calculation, with the following syntactic redefinition:
\bi
\item
Error potentials of 1,2,3,4,5 (in \fig{silapModel}) correspond to issue severity predictions
(from \fig{ff}) of $\_5,\_4,\_3,\_1',\_1'$, respectively.
\ei

\noindent
We call this new version SILAP2.
\subsection{Assessing SILAP2}

SILAP2
can't distinguish between severity leves \_1 and \_2.
When these error potential predictors are applied to \fig{silapModel},
they 
can select slightly more tasks that what is strictly necessary:
in tasks 
2.4, 2.5, 2.6, 4.7, 5.6, 5.8, 5.10  6.1 of
\fig{silapModel}, there are 8 cases
where
the tasks selected by $ep=5$ would not be selected when $ep=4$.
This is not an ideal situation
but, recalling the abherent rules of \fig{5e}, our current data does not support the generation
of good level \_1 predictors.

If budget permits, then all the tasks recommended
by SILAP2 should be applied. However, 
since SILAP2 selects {\em  more} than the minimal IV\& tasks, the research
question becomes how to rank and, possibly, prune the recommended tasks.
\input{tim/ce}

\fig{allocated} shows the available $cost$ per task and the  $frequency$ of performing tasks
 on the current
IV\&V projects.  
For proprietary reasons,
these values are normalized to sum to 100.  The value $\alpha_p$ sums total $cost*frequency$ for each phase $p$.
Following the convetion of \fig{silapModel}, the phases are numbered
\{2,3,4,5,6\}
corresponding to
$concept$, $requirements$, $design$, $implementation$,
$test$ (respectively).

\fig{issues} shows the issues recently found by IV\&V
in 9 robotic missions and 7 man-rated missions. 
In that figure,
the value $\beta_p$ sums the issues found in phase $p$ times a factor that weights high severity
issues exponentially larger than low severity issues.  

An ideal IV\&V tasks finds many issues, early in the lifecyle, at low cost, after
performing just a few tasks; i.e. it maximizes \begin{equation}\label{eq:bg}
\frac{issues\; found\; early}{effort\; expended}
=\frac{\beta_p
\cdot  \Delta^{(6-p)}}{\alpha_p} 
\end{equation}
In this expression, the $\Delta$ term rewards issues found earlier in the lifecycle. 

\fig{phase} is a heuristic assessment of 
performing IV\&V tasks at different phases of the lifecycle. The four results
columns of that figure computer \eq{bg}
for
$\Delta\in\{2,5\}$ (see \fig{dabney06a}) and for the robotics and man-rated mission data of \fig{issues}.
Each column shows the \eq{bg} values as a percentage of the totals for that column.

From \fig{phase}, we conclude that:
\bi
\item
If budgets do not permit performing all the tasks recomended by SILAP2, then late lifecycle
tasks should be pruned away {\em before} concept and requirements phased tasks. Overwhelming,
most of the cost-benefit of IV\&V is seen in the these first two phases.
\item
The relative merits of concept-phase vs requirements-phase IV\&V is crucially dependent on the
cost-to-fix
$\Delta$ value. At $\Delta=2$, requirements IV\&V is more valuable than concepts IV\&V (for both
robotic and manned missions)
but
at $\Delta=5$, the pattern reverses and concept IV\&V is more valuable (particularly for manned missions). 
Organizations should routinely
collect data like \fig{dabney06a} in order to find their own most cost-effective
 phase to perform IV\&V.
\item
Compared to early lifecycle IV\&V, the merits of late lifecycle IV\&V are 
very meger. In \fig{phase} performing IV\&V in the design, implementation, and testing phases
was between 0 to 15\% of the total value. Due to complciations with our data collection,
these percentages have a degree of uncertainty. However, we would defend the general pattern of
\fig{phase}: overwhelmingly, early lifecycle IV\&V is far more valuable than later lifecycle.
\item
\fig{allcosts} sorts all the task costs from \fig{allocated}. Two tasks have an outstandingly
large cost: 5.3 (interface analysis), code; and 6.3 (simulation analysis). Based
on the above, we would argue that these most expensive, late lifecycle, least cost-effective
tasks should only be performed if there is a very pressing business case to do so.
\item
Returning now to the eight problem cases of SILAP2,
we note
that 5 of those cases are late cycle tasks; i.e. can be pruned without degrading much of the value
of IV\&V. 
\begin{figure}
\begin{center}
\includegraphics[width=2in]{allcosts.pdf}
\end{center}
\caption{All task costs from \fig{allocated}, sorted.}\label{fig:allcosts}
\end{figure}
As to the remaing 3 problem cases (the concept phase tasks of 2.4, 2.5, 2.6),
\fig{allcosts} shows that these three are, relatively speaking,
the least costly of all the IV\&V tasks. That is, even if they are selected inappropriately,
they will add little to the total IV\&V cost. 
Hence we assert that the known problems with SILAP2
are neglible. 
\ei
%2007 dollars; and (b)~the tasks are currently on-going.
%The $alpha_i$ 
%
%\fig{allocated} shows 
%required for \eq{value}
%for the same projects. 
%
%
%A heuristic value on IV\&V tasks was sought using:
%\begin{equation}\label{eq:value}
%Value(task) = \frac{Issues\cdot 10^{(5 - Severity)} \cdot \Delta^{(6-Phase)}}{Costs \cdot Frequency}
%\end{equation}
%That is, the most cost-effectve IV\&V tasks are those that we perform at:
%\bi
%\item
% lowest $frequency$;
%\item
%$cost$ the least to complete;
%\item
%while generating the most $issues$, weighted  exponentially by their $severity$ 
%(and severity 1 are most heavily weighted);
%\ei
%Also, the
%term 
%$\Delta^{(6-Phase)}$ gives greater weight to issues found earlier in the lifecycle.
%To understand that term, recall that
%\bi
%\item
%\fig{dabney06a}
%showed that the cost of resolving an issue increased by  a factor $\Delta \in 2..5$ for
%every phase after the issue was created. 
%\item
%\fig{silapModel}
%numbered all IV\&V tasks 2.x, 3.x, 4.x, 5.x, 6.x representing the phases $concept$, $requirements$, $design$, $implementation$,
%$test$ (respectively).
%\ei
%numbered all IV\&V tasks 2.x, 3.x, 4.x, 5.x, 6.x representing the phases $concept$, $requirements$, $design$, $implementation$,
%$test$ (respectively).
%\ei
%Hence, assuming $\Delta=5$,
%\eq{value} values
%a dozen  severity 2 issues found in requirements (phase 3)
%as follows:
%\[
%Issues\cdot 10^{(5 - Severity)} \cdot \Delta^{(6-Phase)} 
%= 12 \cdot 10^{(5 - 2)} \cdot 5^{(6 - 3)}
%\]
%Some extrapolations from the available data was required:
%\bi
%\item
%\fig{allocated} shows the available $cost$ and $frequency$ data on the current
%IV\&V projects.  
%The $cost$ values are estimated, not actual, since (a)~the study wanted to use
%2007 dollars; and (b)~the tasks are currently on-going.
%\item
%The SILAP databases contain 433 sub-systems (only 211 of which were used above since
%only that subset also has associated issue reports). The $frequency$ values in \fig{allocated}
%come
%from passing those 433 sub-systems through SILAP1 to find what tasks would be selected
%by current
%NASA IV\&V practices. That is, these frequency counts relect {\em current}
%NASA IV\&V practice, and not what might happen once SILAP2 is deployed.
%\ei
%\fig{issues} shows the issues recently found by IV\&V
%in 9 robotic missions and 7 man-rated missions. \fig{value} uses this data to
%rank the 
%IV\&V tasks. Seperate rankings were made for:
%\bi
%\item man-rated/Delt
%and robotic
%missions;
%\item the lower and upper bound on the $\Delta$ value from \fig{dabney06a}; i.e. 2 and 5.
%\ei
%
%Several aspects of \fig{issues} are noteworthy.
%
%stablity
%
%many missing
%
%few really really good
%
%research goals. optimize the common. explore the rare.
%
%tracability analysis catch all for XYZ
%
%early lifecyle rules. 
%
%tracaulity rules.
%
%certain uch discssed techniques are absent. eg. formal methods. logn wy abefore such cutting
%edge techniques can be rolled out on a national basis
%
%do all unless limited funding. if limited fhding, at lese do sw.ser
%requreiments allocation analysis .
