\documentclass[journal]{IEEEtran}
%\documentclass{article}
%\documentclass[9pt,journal,compsoc]{IEEEtran}
\newcommand{\aive}{{\"aive}}

\usepackage{pifont}
\usepackage{times}
\usepackage{alltt}
\usepackage{graphicx}
\usepackage{url}
\usepackage{eso-pic}
\usepackage{color}
\usepackage{type1cm}
\makeatletter
  \AddToShipoutPicture{%
    \setlength{\@tempdimb}{.5\paperwidth}%
    \setlength{\@tempdimc}{.5\paperheight}%
    \setlength{\unitlength}{1pt}%
      \makebox(600,900){\rotatebox{45}{\textcolor[gray]{0.80}{\fontsize{9cm}{9cm}\selectfont{Draft}}}}}
\makeatother
%
% IEEE Computer Society needs nocompress option
% requires cite.sty v4.0 or later (November 2003)
%\usepackage[nocompress]{cite}
% normal IEEE
 \usepackage{cite}
 \usepackage{stfloats}

% correct bad hyphenation here

\usepackage{tim/tricks}
\usepackage{ken/tricks}
\usepackage{melissa/tricks}
\usepackage{markland/tricks}
\usepackage{butch/tricks}

\begin{document}

\title{Learning Better IV\&V Practices}
%
\author{Tim~Menzies~\IEEEmembership{Member,~IEEE}%
\thanks{
Dr. Menzies is with the Lane Department of Computer Science and Electrical Engineering,
West Virginia University, USA; \protect\url{tim@menzies.us}},
Markland Benson~\IEEEmembership{Member,~IEEE}% 
\thanks{Mr. Benson is with NASA, formerly at NASA's
IV\&V facility, Fairmont, West Virginia \protect\url{Markland.J.Benson@nasa.gov}},\\
Ken Costello%
\thanks{Mr. Costello is Chief Engineer at NASA IV\&V: \protect\url{Kenneth.A.Costello@nasa.gov}},
Christina Moats% 
\thanks{Ms. Moats is with NASA IV\&V: \protect\url{Christina.D.Moats@nasa.gov}},
Melissa Northey% 
\thanks{Ms. Northey is with NASA IV\&V: \protect\url{Melissa.S.Northey@nasa.gov}},
and
Julian Richardson%
\thanks{Dr. Richardson is with RIACS at NASA Ames \protect\url{julianr@email.arc.nasa.gov}}
\thanks{
See \protect\url{http://menzies.us/pdf/07ivv.pdf} for an earlier draft.}
\thanks{
This research was conducted at
West Virginia University 
under NASA sub-contract project 100005549, task 5e, award 1002193r.
Reference herein to
any specific commercial product, process, or service by trade name,
trademark, manufacturer, or otherwise, does not constitute or imply
its endorsement by the United States 
Government.
}}

\markboth{Journal of ???,~Vol.~X, No.~Y, April~2007}%
{Menzies  \MakeLowercase{\textit{et al.}}: 
Learning Better IV\&V Practices}

\maketitle

%------------------------------------------------------------------------- 
\begin{abstract}
A software independent  verification and validation plan is composed
from
lists of known IV\&V tasks.
Despite two decades of IV\&V research,
making this  plan is difficult.
The literature offers nearly a hundred candidate  tasks.
Clearly, not all tasks are suitable for all projects, but there is little
empirical
evidence about which IV\&V tasks  are most cost-effective.

After data mining NASA IV\&V data,
we offer (a)~a {\em predictor} for project issue severity that can
be applied early in the IV\&V life cycle; (b)~an IV\&V {\em task selector} (that used the predictor
to find the appropriate IV\&V tasks); and (c)~{\em pruning
heuristics} describing what tasks to ignore, if the available budget cannot accommodate
all the selected tasks. 
In 10-way cross validation experiments, the predictor performs very well indeed: the
average $f$-measures
for predicting four classes of issue severity was 0.95.
\end{abstract}
\section{Introduction}
Budgen~\cite{budgen06} challenges software engineering researchers to better document the benefits
of their preferred methods.
In the Internet age, he
argues, many sources of supposed knowledge
(Google, Wikipedia, digg.com,
and so on)
surround us.  He poses
the following challenge: 
``how
should we train students to assess all that information and to
separate the sense from the nonsense?''. In his view, before we can
denounce some inaccuracy in, say, Wikipedia, we must first look to
our own work and audit our own results.  

Accordingly, this article explores what we 
{\em say}, rather that just we can {\em claim},
about software independent verification \& validation (IV\&V). 
In 1989, 
Wallace \& Fugii~\cite{wallace89}  
distinguished between:
\bi
\item {\em Embedded V\&V}, which  is done in-house by the developers or the users; 
\item
And {\em independent V\&V}  
which is done externally by a group that might choose to explore very different
issues to the embedded V\&V team\footnote{Annex C of the IEEE 1012-2004~\cite{1012} 
standard
offers a more detailed definition of IV\&V. That definition retains Wallace \& Fugii's essential 
distinction between internal (embedded) and external (independent) V\&V.}.
\ei

Since 1989,  many organizations have implemented IV\&V
programs including space agencies in Europe, Japan, and America.
NASA's software IV\&V program is one of the
largest: each year, dozens of projects are checked by
an external team.
But identifying cost-effective IV\&V tasks is a difficult task.
We can find nearly 100 such tasks:
\bi
\item
Wallace \& Fugii offer 20 essential and 36 optional
tasks for software V\&V~\cite{wallace89}. 
\item
Table 2 of the  IEEE-1012 standard for software V\&V
offers a list of 52 minimal tasks for IV\&V. Each task is selected by a combination of development phase and
estimated severity in project errors~\cite{1012}.  
\item
NASA's local variant of IEEE-1012 lists the 40 pre-deployment
tasks of \fig{silapModel}.
Each task is selected by an estimate of project {\em error potential} (a.k.a. likelihood)
and {\em consequence} (a.k.a. risk). 
These estimates come from the SILAP1 model, discussed later in this article.
\ei



If budget permits, all these tasks should be performed.
However, in the usual case of limited budgets, software engineers must carefully
select their IV\&V tasks.
Quite possibly, some of those tasks are cheapest to apply while tending to find more issues.
Also, not all tasks are relevant for every project.
But what tasks are best applied to what projects?

\input{tim/silapModel}
\begin{figure*}
\scriptsize
\begin{center}
\begin{tabular}{r|r|r@{~}l|r@{~}l}
\multicolumn{2}{c}{~}&\multicolumn{4}{c}{$F_S$ = number of issues found of severity $S$}\\\cline{3-6}
phase& $S$ = severity& \multicolumn{2}{l}{9 robotic missions}& \multicolumn{2}{|l}{7 man-rated missions
}\\\hline
      2.  concept&                        1&           0&                  & 59& \rule{3mm}{2mm}\\
             &                        2&           7&   & 81&\rule{8mm}{2mm}\\
             &                        3&          54&   \rule{3mm}{2mm}&  168& \rule{9mm}{2mm}\\
             &                        4&          11&   & 0&\\
             &                        5&           2&                  & 0&\\\cline{3-3}\cline{5-5}
             &                     $\beta_2=\sum_{S=1}^5 F_S\cdot 10^{5-S}$    & 12,521           &                  &687,800  & \\\hline
 3. requirements&                        1&           1&                  &  291&\rule{15mm}{2mm}\\
             &                        2&          70&   \rule{4mm}{2mm}&  313&\rule{16mm}{2mm}\\
             &                        3&         874&  \rule{44mm}{2mm}&  430&\rule{21mm}{2mm}\\
             &                        4&         435&  \rule{21mm}{2mm}& 0&\\
             &                        5&         133&  \rule{7mm}{2mm}& 0&\\\cline{3-3}\cline{5-5}
             &                     $\beta_3=\sum_{S=1}^5 F_S\cdot 10^{5-S}$    & 171,883           &                  & 3,266,000 & \\\hline
    4.   design&                        1&           0&                  & 11&\\
             &                        2&          18&   \rule{1mm}{2mm}& 16&\rule{1mm}{2mm}\\
             &                        3&         275&  \rule{13mm}{2mm}& 96&\rule{5mm}{2mm}\\
             &                        4&         129&  \rule{7mm}{2mm}& 0&\\
             &                        5&          25&   \rule{2mm}{2mm}& 0&\\\cline{3-3}\cline{5-5}
             &                     $\beta_4=\sum_{S=1}^5 F_S\cdot 10^{5-S}$    & 46,815           &                  & 135,600 & \\\hline
  5.  implement&                        1&           1&                  & 51&\rule{3mm}{2mm}\\
             &                        2&         59 &   \rule{3mm}{2mm}& 78&\rule{4mm}{2mm}\\
             &                        3&         364&  \rule{18mm}{2mm}& 333&\rule{16mm}{2mm}\\
             &                        4&         282&  \rule{24mm}{2mm}& 0& \\
             &                        5&          76&   \rule{4mm}{2mm}& 0 &\\\cline{3-3}\cline{5-5}
             &                     $\beta_5=\sum_{S=1}^5 F_S \cdot 10^{5-S}$    & 108,296           &                  & 621,300 & \\\hline
   6.      test&                        1&           1&                  & 15 &\rule{1mm}{2mm}\\
             &                        2&          10&   & 27 &\rule{3mm}{2mm}\\
             &                        3&         361&  \rule{18mm}{2mm}& 77 &\rule{4mm}{2mm}\\
             &                        4&         271&  \rule{14mm}{2mm}& 0 &\\
             &                        5&          41&   \rule{2mm}{2mm}& 0 & \\\cline{3-3}\cline{5-5}
             &                     $\beta_6=\sum_{S=1}^5 F_S \cdot 10^{5-S}$    & 58,851           &                  &184,700  & \\\hline
\end{tabular}
\end{center}
\caption{A sample of found issues.}\label{fig:issues}
\end{figure*}
This is an open research question. The state of the research is
typically a study of one task applied to one project. It is rare to find an empirical
study
comparing the
cost-effectiveness  of multiple tasks over multiple projects. 

To answer
 this question,  this study applied  data mining to 40 IV\&V tasks applied
to 
211 NASA sub-systems.
We found that the NASA SILAP1 IV\&V issue severity predictor~\cite{costello05}
has been {\em under-estimating}
the risk of NASA  software for at least the last two years. 
Using
data mining , an update to SILAP1 was created called SILAP2. 

Our data also let us comment on when to best apply IV\&V.
Previously, the benefits of IV\&V have been characterized by a benefit {\em curve} that decreases
as the
life cycle progresses. We offer instead a benefit {\em cliff} were IV\&V is very effective in the concept
and requirements phases, but hardly effective during design, implementation, and test. 


%
%NASA's current IV\&V program. Now, nearly 
%
%One proponent of evidence-based software engineering
%is David Budgen of Durham University. In the Internet age, he
%argues, many sources of supposed knowledge
%(Google, Wikipedia, digg.com,
%and so on)
%surround us.  He poses\footnote{
%Keynote address, CSEET'06. 
%} the following challenge: 
%``how
%should we train students to assess all that information and to
%separate the sense from the nonsense?''. In his view, before we can
%denounce some inaccuracy in, say, Wikipedia, we must first look to
%our own work and audit our own results.  
%
%The time is right for this article.
%Based on NASA's experience,
%two issues will be explored:
%\bi
%\item
%{\em Identifying projects that need more IV\&V:}
%What are the minimum set of project characteristics that most predicts for
%a problem software project?
%\item
%{\em Conducting IV\&V in a cost-effective manner:}
%What are the minimum set of cost-effective IV\&V tasks that most finds specific issues
%in software?
%\ei
%We show that it is possible to build
%effective predictors for projects that
%need more IV\&V.
%{\em Screening experiments} will be conducted,
%that reduce a larger set of factors
%to just those with
%an
%important effect on the severity levels of issues found by IV\&V.
%The simplicity of the predictors will be surprising. Several factors
%that the literature suggests {\em should} predict for
%greater defects are absent from our learned predictors.
%
%We also show that
%the list of most cost-effective IV\&V tasks
%can be readily found.
%This list will be quite small.
%suggesting that many of the  current lists of proposed
%IV\&V tasks (e.g.~\cite{1012,costello05,wallace89}) are
%overly elaborated.
%
%This specific conclusions of this study are two-fold.
%\be
%\item
%NASA has 
%has been consistently {\em under-estimating}
%the risks associated with NASA software projects. 
%\item
%IV\&V is often applied to static code artifacts.  
%We show here the
%analysis of such late life cycle artifacts is one of the least
%cost-effective IV\&V methods.  
%The {\em effectiveness} of early life cycle  V\&V
%has been made before (e.g.\cite{boehm88,dabney06a,shu02,wallace89,arthur99}).
%However, to the best of our knowledge,  this is the first clear result relating to
%the relative {\em ineffectiveness} of later life cycle V\&V.
%\ee
%
%\noindent
%There are two business-level implications of the above:
%
%\bi
%\item
%NASA must expand its IV\&V program.
%\item
%All organizations, not just NASA, should
%{\em reduce} late life cycle verification-based
%IV\&V (that focuses on code artifacts) and {\em increase}
%early life cycle validation-based IV\&V (that focuses
%on the business context of a project).
%\ei
%

%Prior experience with projects resulted in the experience factory approach developed at NASA SEL.
%A premise of that approach was enough organization stability to support a unified, centralized
%reference model for software development.
%That stability does not currently exist at NASA and we cannot expect such stability to appear
%in the foreseeable future. 
%
%However, one of the NASA SEL lessons still applies: 
%data collection  must be compromise in asking for only as much information as is feasible to obtain.
%This study lacked the resources to define a new data collection method. Rather, it relied on the
%data currently available within the databases of the NASA IV\&V
%facility. 
%
%Due to the pace of organizations change with NASA, and within its databases,
%very little of the 15 year track record of NASA IV\&V was available to this project. The best 
%we could do was to assemble 211 records mapping the SILAP1 raw features (collected early in the IV\&V process)
%to the severities of the issues found by IV\&V. This lack of data may seem surprising, but it is quite
%typical of highly dynamic organizations such as NASA.
%
%
%Based on this experience, we offer the following conclusions.
%
%
The rest of this   paper is structured as follows:
\bi
\item
 After documenting
the benefits of IV\&V, a literature review will show
that few papers empirically compare a large range of IV\&V tasks.
\item
In order to collect new data, two alternate approaches will be discussed:
heuristic data mining and a NASA SEL-style ``learning organization'' method.
Given the current pace of change within NASA, this study resorts
to heuristic data mining.
\item
Since it can't collect {\em new} data,
this  study will have to use the {\em available data}, none of which was
collected especially for this purposes. Hence, like many other
real world SE data sources, it contains much noise (signals not connected
to the target). Various noise-reduction methods will be discussed including
RIPPER~\cite{cohen95r} and the WRAPPER~\cite{kohavi97}. 
\item
These methods are then
applied to NASA's current IV\&V task selection model, SILAP1. Drawbacks
with SILAP1 will motivate a new model, SILAP2. 
\item
Empirically, SILAP2 performs very well.
In a 10-way cross validation experiment, SILAP2
is nearly perfect in predicting for issues severity (for four classes of severity,
the $f$ measure was 0.95,0.96,0.95, and 0.94).
\item
In a small number of cases, SILAP2 recommends more IV\&V tasks than what is strictly necessary.
Hence, SILAP2 is augmented with {\em pruning} heuristics 
describing what tasks to ignore, if the available budget cannot accommodate
all the selected tasks. 
\ei

\section{Benefits of IV\&V}


\PARstart{T}{here} is much formal and informal evidence for the value of IV\&V.
By ``informal'' evidence, we mean that evidence not in published sources
that  can be verified by another researcher.
Based on our interviews with NASA IV\&V personnel, we offer the informal evidence
for the benefits of IV\&V:
\bi
\item
``Even when developers assess their own products, they may be focused
on a very small part of a much larger system. IV\&V workers, on the other hand,
can explore larger parts of the system.''
\item
``When faced with schedule pressure,
developers can race to finish a product without taking adequate time
for self-review of that work. These reviews can be conducted by
IV\&V.''
\item
``Historically, it has been observed at NASA
that development teams can change personnel faster than the IV\&V team.
Hence, the IV\&V team can come to understand the software as well or better
than the development team. 
For example, in several NASA multi-year projects, most of the issues
were found
by the IV\&V team, and not the developers.
 Also, in numerous cases, the IV\&V team can recall  why (e.g.)
three years ago, some crucial
decision
was made.
That is, IV\&V has become NASA's corporate memory for software-related
issues.''
\item
``The technical skills of NASA's IV\&V's team are acknowledged by
(some) of NASA's contractors.
In one case, the prime contractor on a NASA project
found funds within its own budget to pay for more IV\&V,
over and above the amount required by their NASA contract.''
\ei

Formal evidence for the benefit of IV\&V comes from the software engineering
literature and NASA technical reports:
\bi
\item
Zelkowitz and Rus discuss the Space Shuttle IV\&V  program in the 1990s~\cite{zelkowitz01}. 
They report a large decrease in major flight defects 
once the IV\&V program was introduced, and   that IV\&V uncovered and resolved
certain
shortcomings in the shuttle's testing program. 
Their most significant finding was that almost half the issues found by
IV\&V were found very early in life cycle of shuttle software upgrades.
\item
\fig{issues} shows the hundreds of  issues found in recent
NASA missions by the IV\&V team. Note that (a)~most of the issues are found early in the life cycle
(in the concept and requirements phase); (b)~a large number of those
issues were of high criticality (severity 1 and 2), especially for the man-rated missions.
\ei
Finding errors early in the life cycle is critical to software cost reduction.
In 1988, Boehm \& Papaccio 
advised that fixing or reworking software is far
cheaper earlier in the life cycle than later ``by factors
of 50 to 200''~\cite{boehm88}.
This effect has been widely documented in other research:
\begin{figure}
\begin{center}
{\footnotesize
\begin{tabular}{r@{~}|r@{~}l@{~}|r@{~}l}
phase & \multicolumn{2}{c|}{IV\&V}&\multicolumn{2}{c}{no IV\&V}\\\hline
requirements& 16&\rule{8mm}{1.5mm}& 0&\\
high-level design& 20&\rule{10mm}{1.5mm}& 2 & \rule{1mm}{1.5mm}\\
low-level design& 31&\rule{15mm}{1.5mm}& 8&\rule{4mm}{1.5mm}\\
coding \& user testing& 24&\rule{12mm}{1.5mm}& 34&\rule{17mm}{1.5mm}\\
integration \& testing& 6&\rule{3mm}{1.5mm} & 14 & \rule{7mm}{1.5mm}\\\hline
totals& 97&& 58 & 
\end{tabular} }
\end{center}
\caption{The Arthur et.al. controlled experiment comparing defects found
with and without IV\&V. From \protect\cite{arthur99}.}\label{fig:arthur}
\end{figure}

\bi
\item
Arthur et.al.~\cite{arthur99} conducted a small controlled experiment
where
a dozen engineers at NASA's Langley Research Center were 
split into development and IV\&V teams. The same application
was written with and without IV\&V. 
\fig{arthur} shows the results:
(a)~more issues were found using IV\&V than without; (b)~the issues
were found much
earlier.
This is a compelling argument for IV\&V. 
Even if the IV\&V team found the {\em same} bugs as the development
team, but found them {\em earlier}, the cost-to-fix for the project
would be reduced by a significant factor.
\item
A panel at IEEE Metrics 2002~\cite{shu02}  debated the general benefit
of early life cycle defect removal. They concluded that
finding and fixing  severe software problems
after delivery is often 100 times more 
expensive than finding and fixing it during 
the requirements and design phase. They panel also commented
that for 
non-severe software 
defects, that factor can be much lower down to a factor of two.
Note that, in either case, it is cheaper to fix earlier than later.
\item
Wallace \& Fugii~\cite{wallace89} discuss 
a 
study where IV\&V was started at different phases.
Starting  at the coding phase
only saved 20\% of the total V\&V costs (embedded + independent).
However,
starting earlier at the requirements
phase saved 92\% to 180\% of the total V\&V costs.
\item
\fig{dabney06a} shows 
the cost of quickly fix an issue relative to
leaving it for a later phase (data from four NASA projects~\cite{dabney07}). 
The last line of that table shows that
delaying issue resolution 
even by one  phase
increases the cost-to-fix of $\Delta=2 \ldots 5$.
Using this data, 
Dabney et.al.~\cite{dabney07} calculated that 
every dollar
spent on IV\&V 
returned to NASA \$1.21, \$1.59, \$5.53, and \$10.10 (on four NASA projects). 
\ei



\begin{figure}
\begin{center}
{\scriptsize 
\begin{tabular}{r@{~}r|r@{~}r@{~}r@{~}r@{~}r@{~}r}
\multicolumn{2}{c}{~}&\multicolumn{6}{c}{Phase issue found}\\\cline{3-8}
&Phase issue        & f=1        & f=2   & f=3  & f=4 &f=5 &f=6\\
i&introduced   &Requirements& Design& Code &Test &Int& Operations\\\hline 
1&Requirements &           1& 5     & 10   & 50  & 130& 368\\
2&Design        &            &  1   & 2    & 10  &  26&  74 \\
3&Code          &            &      & 1    & 5   & 13 &  37\\ 
4&Test           &           &      &      & 1   & 3 & 7\\ 
5&Integration    &          &       &      &     & 1 & 3\\\hline
   &$\Delta$ = mean $\left(\frac{C[f,i]}{C[f,i-1]}\right)$
                   &&5          & 2    & 5   &2.7     &2.8\\            
\end{tabular} }
\end{center}
\caption{$C[f,i]$: The cost-to-fix escalation factors, relative to
fixing an issue in the phase where it was found ($f$) versus
in the phase where it was introduced ($i$).
Last row shows the cost-to-fix delta if the issue  introduced in phase $i$ 
is fixed immediately afterwards in phase $f=i+1$. From~\cite{dabney07}.}
\label{fig:dabney06a}
\end{figure}

\section{Related Work}
\PARstart{H}{aving} documented the benefits
of IV\&V, the next question a manager might ask is ``how to do it?''.
This is a difficult question to answer.
In the literature, there are very few attempts to empirically assess the relative
cost-effectiveness of a broad range of IV\&V techniques.

Most of the reports
take the form of (e.g.):
\bi
\item
Easterbrook et.al.~\cite{easter98} who discuss the merits of using a single
lightweight formal method for IV\&V 
requirements models on one NASA project;
\item Hayes, Dekhtyar, and Sundraram who explore
the relative merits of
different tools for finding links between requirements documents~\cite{hayes06}. That analysis includes a comparison
of human expert and state-of-the-art commercial tools and a new research prototype.
\ei
(For more examples of singleton evaluations of one method on one project,
see
the proceedings of the annual
symposium 
Program\footnote{\url{http://sas.ivv.nasa.gov/conclusions/index.php}} and
and repository of NASA's Software Assurance Research\footnote{\url{http://sarpresults.ivv.nasa.gov}}.)


 Several researchers offer detailed taxonomies of standard NASA 
faults~\cite{hayes05,malin07},
but
do not address what tasks are most cost-effective at finding the faults. 
For example,
Lutz reports
studies on the most frequent kind of faults seen in deep space NASA missions but does
fully address the issue of what V\&V and IV\&V tasks are best at removing faults~\cite{lutz04}. 

Leveson talks in general terms
about how to increase the safety of software~\cite{leven95}
 but since her preferred methods would replace
current IV\&V tasks~\cite{levenson96}, it is difficult to map that advice into
IV\&V task selection.

In a limited number of publications, some attempt is made to generalize over
multiple projects, multiple methods, or projects extending over many years.
Those publications include:
\bi
\item
Madachy's  heuristic software risk model that offers alerts when certain pairs of
observations are seen in a project~\cite{madachy97};
\item
Boehm and Basili's ``top 10'' list of defect reduction methods~\cite{boehm01,shu02};
\item Dabney et.al.'s study, discussed above, based on four NASA projects~\cite{dabney07};
\item Menzies et.al.'s discussion of 
\bi
\item
the merits of different data miners to learn predictors for software defects
using data from five NASA projects~\cite{me03q,me07b}.
\item
methods for generating cost models from 25 software projects, including 11
 NASA developments~\cite{me06d};
\ei
\item
Glass' list of six factors seen in 16 ``runaway'' software projects;
i.e. projects where
``schedule, cost, or functionality that was twice as bad as the original estimates''~\cite{glass97}; 

\item Raffo et.al.'s software process of a standard waterfall model, modified to handle IV\&V tasks~\cite{raffo04}.
\item
Jiang et.al.'s
table of the connection of
40 project features to the software risks seen in 83 projects~\cite{jiang02}.
\item
Ropponen \& Lyytinen
analysis of questionnaires from 83 project managers and 1,110 projects that identifies
26 software risk components~\cite[p110]{ropp00};

\item
Takagi \& Abe et.al.'s study of 22 project features from 31 projects 
that identifies features that predict
for software ``runaways''~\cite{takagi05,abe06}
\ei
\begin{figure*}
\scriptsize
\begin{center}
\begin{tabular}{r|r@{~}l|p{2.75in}|l}
year & & \#IV\&V project at Fairmont IV\&V &   notes & oversight\\\hline
1988 &n/a & &Space shuttle begins IV\&V&\\
1991 &n/a& &WVU awarded grant to build Fairmont IV\&V& NASA headquarters (east  coast)\\\hline
1994 &1  &\rule{1mm}{1.5mm} &International space station IV\&V at Fairmont begins &NASA AMES (west coast)\\
1995& 1 &\rule{1mm}{1.5mm}&&\\
1996& 2 &\rule{2mm}{1.5mm}&&\\
1996& 3 &\rule{3mm}{1.5mm}&New contracts enables IV\&V for all NASA projects
        (but projects must fund IV\&V from their own line items) &\\
1997& 3 &\rule{3mm}{1.5mm}&&\\
1998& 3 &\rule{3mm}{1.5mm}&&\\
1999& 12 &\rule{12mm}{1.5mm}& NASA mandates IV\&V for all NASA software&\\\hline
2000& 15&\rule{15mm}{1.5mm}& Facility oversight moves to Goddard Space Flight Center& GSFC (east coast)\\
2001& 20 &\rule{20mm}{1.5mm}&&\\
2002& 36&\rule{36mm}{1.5mm}&&\\
2003& 42 &\rule{42mm}{1.5mm}&IV\&V now funded from a central agency source. &\\
2004& 37&\rule{27mm}{1.5mm}&&\\
2005& 24&\rule{24mm}{1.5mm}&SILAP1 data collection begins&\\
2006& 26&\rule{26mm}{1.5mm}&&\\
2007& 24&\rule{24mm}{1.5mm}&&\\
\end{tabular}
\end{center}
\caption{History of NASA IV\&V.}\label{fig:hist}
\end{figure*}

For our purposes, these studies are incomplete:
\bi
\item
Some of this research bases its conclusions on a handful of projects~\cite{glass97,dabney07,me03q,me07b};
\item
Others use a Delphi-style analysis
 were the authority of the conclusions
rests on the authority of the author~\cite{boehm01,shu02,madachy97}
Alternatively, they may
base their conclusions on extensive data collection
but, for reasons of corporate confidentiality, can not
make that data available for public scrutiny~\cite{jiang02,ropp00,raffo04}.
It is hard to critical audit conclusions  based on a Delphi-style analysis or
inaccessible data.
In general, only a minority of SE researchers are permitted
to publish the corporate data used to make their conclusions (for example,~\cite{me06d,takagi05,abe06,me03q,me07b}).
\item
Several of the these papers~\cite{jiang02,ropp00} offer general advice about how to avoid problems in software development.
The subjective nature of this advice makes it difficult to consistently
deploy
them 
over a national software development program.
Only a few~\cite{me06d,me07b,abe06,takagi05}. 
offer public domain versions of their data and models.
\item
Worst of all, even when taken in combination, these publications cover just a small
fraction of the numerous V\&V and IV\&V tasks listed in the literature~\cite{wallace89,1012,costello05}.
\ei

\section{Data Mining}

\PARstart{W}{hen} existing sources fail to inform, analysts can turn to new sources.
This study applied  data mining to 40 IV\&V tasks applied
to 
211 NASA sub-systems from 6 projects.

Data mining can be characterized as the heuristic exploration data of the available data.
That data is expected to be noisy (i.e. contain signals not related to the target concept)
so various methods are applied to scrub the data and prune superfluous parts of the learned
model. 

An alternate strategy, that avoids noisy data, is to design a data collection
program that is certain to only collect consistent data relating to a specific business goal.
This section explains why this alternate strategy was not used here.

\subsection{An Alternative Method}

Prior work at the NASA software engineering laboratory (SEL) argued for the careful
design of metrics in order to answer questions relating to significant business 
goals~\cite{basili02,basili88,basili99}.
In this ``learning organization'' 
a {\em quality improvement paradigm} is applied the scientific method to software engineering to
create an
{\em experience factory} 
that systematically saved and reused experience from previous projects.  

The SEL's learning organization experiment lasted ten years (from 1984 to 1994)
during a
period of relative stability within the NASA organization. 
Starting in 1995, the pace of change within NASA increased.
New projects were often out-sourced and the SEL became less the
driver and more the observer, less pro-active and more reactive
to software development activities. Without an owner of
the software development process, each project could adopt its
own structure. SEL-style experimentation became difficult since
there was no longer a central model to build on.
Basil et.al.~\cite{basili02} describes this 1995-2001 phase as one
of ``retrenchment'' at the SEL.

The pace of change that was so fatal to the SEL's learning organization
has not decreased:
\bi
\item
In the 1990s, NASA moved to  a ``faster, better, cheaper'' paradigm that
led to massive changes in NASA's development practices. For example, prior to 1990, NASA's Jet Propulsion
Laboratory launched (on average) five spacecraft a year, all of which were built internally. In the 1990s,
JPL launched dozens of craft, most of which were built by external contractors. 
\item
This policy led
to certain high profile errors which were attributed to production haste, poor communications, and
mistakes in engineering management~\cite{davis00}. Accordingly, NASA terminated  Faster, Better Cheaper 
and changed development methods. 
\item
In 2003, the loss of the Challenger shuttle during re-entry prompted
another major agency reorganization. Development budgets were significantly altered to fund the ``return to flight''
program that led to the successful relaunch, in 2005, of the shuttle program.
\item
As the time of  this writing, NASA is significantly reorganizing  (again) around the ``mission to Mars''.
\ei


NASA's numerous national reorganizations  have been reflected in the
IV\&V program.
\fig{hist} shows that
three times (in 1991, 1994, 2000),
the oversight for IV\&V has changed
from NASA headquarters then to two centers on different
sides of the country.
Also,
in 1996, 
NASA developed new contractual methods that allowed IV\&V to be
applied to a wide range of other projects.
These new methods resulted in  large change to IV\&V management practices
as the focus expanded from a few on-going projects to dozens of smaller
projects.

In 1999, after 11 years 
of success,
IV\&V was mandated
for the entire NASA organization. 
But a problem remained-
projects had to find funds for IV\&V from
within their own budgets. This  often slowed IV\&V
work since
managers were forced to juggle the demands of
developments with the demands of IV\&V.
It took four more years of successful IV\&V before NASA
could solve this problem.
In 2003, NASA reaffirmed and strengthened its support for IV\&V
allowing
projects
to fund their IV\&V from a headquarters'  source.

The above changes meant that more projects could receive the benefits of IV\&V.
However, the same changes that benefited NASA also
complicated
the twin goals of
data collection and analysis.
In nearly all projects,
the schema of IV\&V's issue tracking
systems was 
modified to better address some specific requirement.
While some data fields were required,
most 
were not suitable for comparing
the cost-effectiveness of IV\&V across projects.

In summary:
\bi
\item
  NASA is similar to many
other large organizations that constantly adapt their practices
and their databases as their
business changes. 
Where possible, database schemas are standardized across multiple projects and multiple years.
However,  NASA's record keeping is in as much flux as the rest of
the organization and no data is collected consistently across all projects and all years.
\item
Due to the pace of organization change,
this study could not use
the learning organization  approach pioneered at the
NASA SEL.
The stability required for such an analysis is missing.
\ei
This study had to use the available data, none of which was collected especially
for this purpose. Hence, like many other real world SE data sources,
it contains much noise.

\subsection{Handling Noisy Data}

When learning from noisy sources, it is important to use methods that aggressively
prune noisy signals. 
Two such methods were used here:
\bi
\item
The RIPPER~\cite{cohen95r} rule learner;
\item
The WRAPPER algorithm~\cite{kohavi97} 
that explores subsets of the
available features. 
\ei

Both methods are standard techniques for removing noise and
generating succinct, easily explainable,
models from data.
Other data mining methods were not employed since they are known to generate
superfluous redundancies.
For example, tree learners such as C4.5~\cite{quinlan92}
or CART~\cite{breiman84}
execute in local top-down
search, with no memory between different branches.  Hence, the same concept
can be needlessly repeated many times within the tree.
Consequently, such trees can be  cumbersome, needlessly large, and
difficult to understand. 

RIPPER, on the other hand,
takes an aggressive approach to pruning away superfluous parts of a model:
\bi
\item 
After building a {\em rule}, RIPPER performs a back-select to see what rule
conditions
can be deleted, without degrading the performance of the rule.
\item
Similarly,
after building a {\em set of rules}, RIPPER performs a back-select to see what {\em rules}
can be deleted, without degrading the performance of the rule set.
The learned rules are built while minimizing  their {\em description length}.
This is an information theoretic measure computed from the size of the learned rules,
as well as its errors. If a rule set is over-fitted, the error rate increases,
the description length grows, and RIPPER applies  rule set pruning.
\item
RIPPER also employs a novel
{\em rule set optimization} algorithm that tries
replacing rules
with some straw-man alternatives (i.e. rules grown very quickly).
\ei

RIPPER is one of the fastest rule learners currently known in the literature. In its
haste, it may miss parts of the model that can be pruned. Hence, a much slower
and more thorough feature pruner was also employed:
\bi
\item 
Kohavi \& Johns' WRAPPER algorithm~\cite{kohavi97}
explores subsets of the available features. Each subset is assessed by
asking some target
learner (in our case, RIPPER)
to build a model using just that subset.
\item
The WRAPPER grows the feature
set whenever the target learner reports that the larger set does no better than the subset.
\item
WRAPPER stops when there are no more variables to
select, or there has been no significant improvement in the learned
model for the last five additions (in which case, those last five
additions are deleted). Technically, this is a hill-climbing forward select
search with a ``stale'' parameter set to 5.
\item
In empirical comparisons with other feature pruners, Hall \& Holmes 
report that WRAPPER is  slow, but yields the best results~\cite{hall03}.
\ei
\input{tim/severity}

\subsection{Modeling the Dependent Feature}

RIPPER inputs ``feature vectors''; i.e. sets of independent features plus one
dependent feature called the {\em class}. The goal of the learning is to find
some combination of the independent features that predict for the dependent class feature.

In this study, the dependent feature was the number of issues found in a sub-system,
weighted by the severity of that issue.
NASA now uses a standard five-point scale  to score issue severity.  A
slightly different scale is used for robotic and human-rated missions
(see \fig{robots} and \fig{humans}) but the interpretation of the
scales is same across all missions: severity one issues imply total
disaster while severity five issues are ignorable.

For the purposes of analysis,
the severity counts were summarized 
as follows. Let $s_{i,j}$ be the frequency of severity issue $i$ in sub-system
$j$. If $Max_i$ is the maximum number of issues of severity $i$, totaled
across all sub-systems, 
then $W_j$ is a weighted sum
where all severity one issues are multiplied by  $10^5$, severity two issues
are multiplied by $10^4$, and so on:
\begin{equation}\label{eq:w}
W_j= 1 + \sum_{i=1}^5\left( \frac{s_{i,j}}{1+Max_i}{\cdot}10^{6-i}\right)
\end{equation}
Note that:
\bi
\item
The severity counts were divided by the maximum severity counts so that any
severity $i+1$ error out-weighed any severity $i$ error. This models the business
intuition that (e.g.) a single severity one issue was more important than any number
of severity two errors.
\item
The 
``plus one'' parts of this equation were added to avoid numeric errors. 
In this data, the severity frequency counts were
large enough to make the ``plus one'' contributions negligible.
\ei

\begin{figure}
{\footnotesize
\begin{tabular}{|r@{~=~}p{3.2cm}|r@{~=}p{3.2cm}|}\hline
\multicolumn{2}{|c|}{Derived}&
\multicolumn{2}{|c|}{Raw features}\\\hline
	co&Consequence&

am&Artifact Maturity\\

	dv&Development&


as&Asset Safety\\

	ep&Error Potential& 

cl&CMM Level\\
	pr&Process&
cx&Complexity\\
	sc&Software Characteristic& 
di&Degree of Innovation\\

\multicolumn{2}{|c|}{~}&do&Development Organization\\
\multicolumn{2}{|c|}{~}&dt&Use of Defect\newline  Tracking System\\
\multicolumn{2}{|c|}{~}&	ex&Experience \\
\multicolumn{2}{|c|}{~}&	fr&Use of Formal Reviews\\
\multicolumn{2}{|c|}{~}&	hs&Human Safety\\
\multicolumn{2}{|c|}{~}&	pf&Performance\\
\multicolumn{2}{|c|}{~}&	ra&Re-use Approach\\
\multicolumn{2}{|c|}{~}&	rm&Use of Risk\newline Management System\\
\multicolumn{2}{|c|}{~}&	ss&Size of  System\\
\multicolumn{2}{|c|}{~}&	uc&Use of CM\\
\multicolumn{2}{|c|}{~}&	us&Use of Standards\\\hline
\end{tabular} }
\caption{The SILAP1 features. Project identifies are shown top
left. Derived features are shown bottom left- and these
are derived from the 16 raw features shown on the right hand side.}\label{fig:silapFeatures}
\end{figure}


\subsection{SILAP1: Modeling the Independent Features}
A learned model predicts for the dependent feature using some combination
of independent features. The independent features used
in this study came from NASA's SILAP1 model~\cite{costello05}.
(SILAP= {\em Software Integrity
Level Assessment Process}) 

From 2005, it was required
that a standard set of project features
are collected at the start of every IV\&V project. 
The features of this SILAP1 model
were identified
via
extensive 
Delphi sessions amongst NASA civil servants in 2005.
Predictors were written for error potential (a.k.a. likelihood) and consequence of error (a.k.a. risk).
These predictors held
the group's consensus belief on effects the quality of NASA projects.
\fig{silapFeatures} shows the
SILAP1 ``derived'' factors that are computed from the ``raw'' factors using the functions of \fig{SILAP.awk}.
These computed error potentials and consequences were then used to select IV\&V tasks, using
\fig{silapModel}.

 SILAP1 grew into an early life cycle project assessment methodology for IV\&V.
In operation, it takes two weeks (or more) full-time work to determine
values for the raw features of \fig{silapFeatures}.
In that two week period, IV\&V analysts
review project documents;
offer a score for each SILAP1 feature;
and write a detailed rationale for that score.
Most of  the SILAP1 features are scored low to high, 1 to 5 (respectively)
or ``0'' for ``don't know''.  

NASA civil servants at IV\&V
report that SILAP1 has greatly clarified their discussions
with projects regarding what IV\&V tasks are/are not to be performed.
One of the advantages of SILAP1 is that, since it was built locally at
IV\&V, it is
understood locally.
Previous risk models, which had been developed elsewhere,
were to a degree ``black box'' models that were 
poorly understood. Hence, it was harder to defend their conclusions
to clients lobbying
for less IV\&V oversight.

Each raw SILAP1 factor is weighted and, usually, filtered by a second numeric to derive
the consequence and error potential values.
The {\em weight*filter} column of \fig{cons} shows the relative contribution of each raw
factor, as found by the Delphi sessions amongst the NASA civil servants.
The following study is the first experimental
verification of the correctness of those
relative contributions.
\begin{figure}[!t]
{\scriptsize
\begin{alltt}
function CO( tmp) { 
        tmp=0.35*AS + 0.65 *PF; 
        return (round((HS) > tmp) ? HS : tmp) } 
function EP() {table
        return round(0.579*DV() + 0.249*PR() + 0.172*SC()) } 
function SC() { 
        return 0.547*CX + 0.351*DI + 0.102*SS } 
function DV() { 
        return 0.828*EX + 0.172*DO } 
function PR() { 
        return 0.226*RA + 0.242*AM + formality() } 
function formality() { 
        return 0.0955*US + 0.0962*UC + 0.0764*CL  +
               0.1119*FR + 0.0873*DT + 0.0647*RM }
\end{alltt}}
\caption{SILAP.awk: computes the derived features from the raw features of \protect\fig{silapFeatures}.}\label{fig:SILAP.awk}
\end{figure}
\begin{figure}
\begin{center}
\footnotesize
\begin{tabular}{p{1.2cm}|rrrr|r}
& & & & weight*& contribution\\
goal & feature & weight & filter & filter & to goal\\\hline
    consequence&      hs&   0.35&      1&0.350& 35\%\\
     (co)&      pf&   0.65&      1&0.650& 65\%\\\hline
     error&      ex&  0.828&  0.579&0.479& 47\%\\
     potential&      cx&  0.547&  0.172&0.094&  9\%\\
     (ep)&      do&  0.172&  0.579&0.100&  9\%\\
     &      di&  0.351&  0.172&0.060&  6\%\\
     &      am&  0.242&  0.249&0.060&  6\%\\
     &      ra&  0.226&  0.249&0.056&  5\%\\
     &      us& 0.0955&  0.249&0.024&  2\%\\
     &      uc& 0.0962&  0.249&0.024&  2\%\\
     &      fr&  0.119&  0.249&0.030&  2\%\\
     &      dt& 0.0873&  0.249&0.022&  2\%\\
     &      ss&  0.102&  0.172&0.018&  1\%\\
     &      cl& 0.0764&  0.249&0.019&  1\%\\
     &      rm& 0.0647&  0.249&0.016&  1\%
\end{tabular}
\end{center}
\caption{Relative contribution of \protect\fig{SILAP.awk} factors to the goals of consequence and error potential.
The percentages shown on the right are normalized values for the $weight*filter$ column. }\label{fig:cons}
\end{figure}
\begin{figure}
\begin{center}
\includegraphics[height=2.25in]{tim/coepissues3D.pdf}
\includegraphics[height=1.5in]{tim/coepissuesCOW.pdf}
\includegraphics[height=1.5in]{tim/coepissuesEPW.pdf}
\end{center}
\caption{SILAP1's computed error potentials (ep), consequence (co), and weighted severities count (W) from
\protect\eq{w} seen in 211
NASA sub-systems from 6 systems. The bottom two plots are 2D projections of
the top 3D plot.}\label{fig:epcow}
\end{figure}

\section{Problems with SILAP1}

This study could access
SILAP1 descriptions of 280 sub-systems from 11 NASA projects. All these projects
received IV\&V in the period 2005 to 2007.
Of these, only   six of those  projects could be mapped into the issue tracking report
system.
Therefore this study was based on data from 211 sub-systems taken from 6 projects.

Each sub-system was described  using:
\bi
\item
the system name: $L1$;
\item
 the sub-system type: $L2$. These types include GNC (guidance and navigation);
PAYLOAD (specialized
payloads); CDH  (command \& data handling), and GROUND (ground systems).
\item
the 16 raw SILAP1 features of \fig{silapFeatures};
\item
and five counts $s_1,\ldots s_5$ showing
the number of issues at differing severity levels. 
\ei
 In this data,
the frequency of issues, relative to issue 1 
issues, was 1:24:206:127:23 for $s_1, s_2, s_3, s_4,s_5$ issues, respectively.
That is, level 3 issues were the majority and level 1 issues were relatively rare.



\fig{epcow} shows the relationship between the outputs derived
by SILAP1 (from \fig{SILAP.awk}) and the weighted sum $W$ of the severity reports.
If SILAP1 was an accurate predictor for problems, we would expect some correlation between
$W$ and {\em either} 
consequence of error (co) {\em or} error potential (ep). A visual inspection suggests that
neither correlation exists.

\begin{figure}
\definecolor{gray}{gray}{0.8} 
\definecolor{g}{gray}{0.81} 
\scriptsize
\begin{center}
\begin{tabular}{r@{~}|r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r}
&as&pf&ex&do&us&uc&cl&fr&dt&rm&ra&am&cx&di&ss&$W$\\\hline
hs&24&18&4&12&-10&34&-20&43&44&3&7&13&8&46&21&-21\\
as &&48&-20&-7&-25&-4&-30&16&24&-10&0&-7&42&23&9&-25\\
pf & &&-9&-16&-21&-5&-26&5&19&1&13&-2&44&11&12&-11\\
ex & & &&-12&\colorbox{gray}{73}&29&\colorbox{gray}{67}&-27&-3&-28&-3&30&-29&33&29&\colorbox{gray}{68}\\
do & & & &&6&9&4&30&-20&37&-23&4&-27&2&-23&-3\\
us & & & & &&37&\colorbox{gray}{58}&-25&-4&-21&-22&11&-43&24&7&\colorbox{gray}{56}\\
uc & &  & & & &&37&\colorbox{gray}{51}&41&13&-3&47&3&30&7&-18\\
cl & & & & & & &&-26&-28&-7&-6&\colorbox{gray}{53}&-25&10&23&\colorbox{gray}{65}\\
fr & & & & & & & &&35&32&31&0&20&12&-21&\colorbox{gray}{-66}\\
dt & & & & & & & & &&2&-8&-14&35&31&23&-37\\
rm & & & & & & & & & &&28&23&2&-20&-30&-18\\
ra & & & & & & & & & & &&14&28&-7&11&-1\\
am & & & & & & & & & & & &&4&12&14&17\\
cx &  & & & & & & & & & & & &&4&27&-33\\
di & & & & & & & & & & & & & &&31&13\\
ss & & & & & & & & & & & & & & &&33\\
\end{tabular}
\end{center}
\caption{Percent 
correlations within 211 sub-systems
between SILAP1 parameters scored 1,2,3,4,5 and the $W$ score from \protect\eq{w}.
The nine pairs with \mbox{$correlation \le -50\%$ or  $correlation \ge 50\%$} are highlighted
in a \colorbox{gray}{gray box}.
}\label{fig:corr}
\end{figure}

\fig{corr} raises other doubts about the efficacy of SILAP1.
Correlation\% varies from -100\% to 100\% and a correlation of zero means that there is no numeric
correspondence between two variables.
As shown in \fig{corr},
most of the correlations in our data have correlations close to zero.
In terms of building succinct models, this is a useful result since it suggests that
a predictor for issue severity might be built using just a small subset of the SILAP1 features.
We will return to this point, below.

In only nine cases (those shown with gray boxes in \fig{corr}) 
are the correlations above 50\% or below -50\%.
Some of those correlations corresponds to established wisdom in the field. For example,
Basili \& Shull \& Lanubile~\cite{basili99}
would accept the strong correlation between increasing
the use of formal reviews (fr) and decreasing high severity issues (correlation=-66\%).
However, we know of no researcher who would accept the following adherent results:
\bi
\item
Increasing programmer experience (ex) 
{\em increases} high severity scores ($W$), correlation=68\%.
\item
Increasing CMM-level (cl) {\em increases} high severity scores ($W$), correlation=65\%.
\ei

A review of our data  explains these adherent results. Recall that
\eq{w} weighted severity one errors by a factor of 100,000. Hence,
even a small system with just a few sub-systems
can have a disproportionate impact on the correlations
if that system has severity 1 issues.
This is indeed the case in our data.
One system, with relatively few sub-systems had most of the
severity one issues. The managers of that system
were highly concerned about the
quality of their product. Hence, they employed numerous best practices
including high CMM practices (cl), extensive use of standards (us),
hiring very experienced developers (ex). That is, these practices did not
{\em cause} the severity one issues in that system. Rather they were adopted
{\em in response} to the presence of those issues. 

 
Based on the above, we  comment that:

\bi
\item
\fig{epcow} shows that
SILAP1's calculated consequence and error potential values are poor predictors for
issue severity.
Hence, SILAP1 needs revision. 
\item
\fig{corr}
shows that only a subset of the SILAP1's features are connected to other features.
Hence, 
a predictor for issue severity might use only a subset of the
SILAP1's features. 
\item
The percentage of level 1 severities in our sample is so small that quirks in
a single system can lead to adherent results. Hence, it is unlikely that we
will be able to build effective predictors for level 1 severity issues. Accordingly,
in the sequel
we will experiment with combining level 1 and level 2 issues into a single ``worst case''
target class.
\ei

\section{Learning SILAP2}

\subsection{Pre-Processing}

RIPPER's rules predict for discrete classes so, prior to learning, the \eq{w} results
for the 211 sub-systems were sorted and divided into {\em bins} with equal population size
(equal frequency discretization~\cite{dou95}). The \fig{corr} results
showed some adherent results resulting from a small number of projects with level 1 severities.
Accordingly, the learning experiments were conducted twice:
\bi
\item
Once where the \eq{w} values were divided into five target classes: \_1, \_2, \_3, \_4, \_5
representing the bins containing (left to right) the most to least high severity issues.
\item
Once where the \eq{w} values were divided into four target classes, generated as follows.
After applying five-binning, the two bins representing severity 1 and severity 2 issues
were  combined  into one. In the four class system, we denote 
the combined class as
$\_1'=\_1 \cup\_2$. 
\ei
\begin{figure}
\begin{center}
\footnotesize
\begin{tabular}{r|rr|r}
&\multicolumn{2}{c|}{number of classes}\\\cline{2-4}
feature	&4&	5	&average\\\hline
us&		10&	10&	10.0\\
am&	    9&	9&	9.0\\
ex&		8&	9&	8.5\\
rm&		6&	9&	7.5\\
ra&		5&	5&	5.0\\
ss&	    5&	4&	4.5\\
fr&		6&	1&	3.5\\
uc&		6&	1&	3.5\\
do&		3&	3&	3.0\\
cl&		4&	2&	3.0\\
dt&		2&	&	2.0\\
pf&		2&	&	2.0\\
cx&		2&	2&	2.0\\
di&		2&	1&	1.5\\
as&		2&	1&	1.5\\
hs&		1&	1&	1.0
\end{tabular}
\end{center}
\caption{Number of times a feature was selected in 10y
experiments where features were assessed using 90\% of
the data (selected at random).}\label{fig:fss}
\end{figure}

\begin{figure*}[!t]
\footnotesize
\begin{center}
\begin{tabular}{r|cr|rrrrr|rl}
&&\multicolumn{1}{c}{~}&\multicolumn{5}{c}{$f$ measure for different classes}\\\cline{4-8}
treatment&number of classes&features&\multicolumn{2}{l}{$\_1'=\_1\cup \_2$}&\_3&\_4&\_5&F&\\\hline
4a&4&L2&0.57&&0.00&0.31&0.00&0.22&\rule{7mm}{2mm}\\
4b&&us&0.64&&0.60&0.00&0.00&0.31&\rule{10mm}{2mm}\\
4c&&L1&0.67&&0.69&0.00&0.46&0.46&\rule{15mm}{2mm}\\
%4d&&l1+l2&0.96&0.89&0.94&0.68&&0.87&\rule{30mm}{2mm}\\
4d&&all 16 raw SILAP1 features&0.93&&0.95&0.87&0.90&0.91&\rule{30mm}{2mm}\\
4e&&us+am+ex+uc+fr+rm&0.95&&0.94&0.97&0.92&0.94&\rule{31mm}{2mm}\\
4f&&ex+us+am&0.95&&0.96&0.955&0.94&0.95&\rule{32mm}{2mm}\\\hline
\multicolumn{10}{c}{~}\\
treatment&number of classes&features&\_1&\_2&\_3&\_4&\_5&F&\\\hline
5a&5&L2&0.00&0.46&0.00&0.31&0.00&0.15&\rule{5mm}{2mm}\\
5b&&L1&0.00&0.54&0.69&0.00&0.46&0.34&\rule{11mm}{2mm}\\
5c&&us&1.00&0.58&0.60&0.00&0.00&0.44&\rule{15mm}{2mm}\\
%5d&&l1+l2&1.00&0.94&0.86&0.94&0.72&0.89&\rule{30mm}{2mm}\\
5d&&all 16 raw SILAP1 features&0.97&0.94&0.93&0.92&0.94&0.94&\rule{31mm}{2mm}\\
5e&&us+ex+am+rm&0.97&0.95&0.96&0.94&0.93&0.95&\rule{32mm}{2mm}
\end{tabular}
\end{center}
\caption{$f$ measures (from \eq{f})
and $F$ measures (from \eq{F}) for different learning treatments.
Note that the \_1' class of the four class system
contains the union of all the \_1 and \_2 classes of the five class
system. }\label{fig:ff}
\end{figure*}

\subsection{Feature Pruning}

\fig{fss} shows the WRAPPER results for four and five target classes.
The WRAPPER was run ten times using RIPPER as the target learner.
Each run used a randomly selected 90\% sample of the data.
As predicted by the \fig{corr} results,
best predictors were found using a small subset of the RIPPER features.
In the four class
experiment:
\bi
\item Use of standards (us) was always selected.
\item
Artifact maturity (am) and developer experience (ex)
were often selected (8 times or more). 
\item
Use of formal reviews (fr), use of configuration
management (uc), and use of a risk management system (rm) were selected six times.  
\ei
In the five class experiment, fewer features were selected:
\bi
\item
Use of standards (us) was found in all ten runs.
\item
Artifact maturity (am), developer experience (ex), and use of a risk management system (rm)
were selected nine times out of ten.
\ei

\fig{fss} raises further doubts about SILAP1. Recall from \fig{cons} that Delphi sessions amongst NASA civil
servants had ranked the contribution of the raw features to the goals of computing error potential 
and consequence. \fig{cons} agrees with \fig{fss} 
that developer experience (ex) is an important predictor for error potential. However, \fig{fss}
significantly differs from \fig{cons} on the other features. Features like $us$ and $am$
were ranked very low by the Delphi session, but
turn out to be quick powerful predictors (at least, when used by RIPPER).





Experiments were then conducted with RIPPER, using various feature subsets:
\bi
\item 
The subsets of selected features found in the four and five class feature pruning experiments
of \fig{fss}. 
\item
All the {\em raw} SILAP1 features. 
\item
Just the system name: $L1$. 
\item
Just the sub-system type: $L2$. 
\ei
\subsection{Performance Measures}

The results were assessed using $F$; i.e. the  average f-measure
seen in all the target classes. 
$F$ is defined as follows.
Let $\{A_x,B_x,C_x,D_x\}$ denote the
true negatives, 
false negatives, 
false positives, and 
true positives
(respectively) for class $x$.
Various performance measures can be computed from these values. 
For example, recall (or $pd$) comments on how much of the target was found.
\begin{equation}
pd_x=recall_x=\frac{D_x}{B_x+D_x}
\end{equation}
Also, precision (or $prec$) comments on how much of the instances that
triggered the detector actually containing the target concept.
\begin{equation}
prec_x=precision_x=\frac{D_x}{D_x+C_x}
\end{equation}
The $f$-measure is the harmonic mean of precision and
recall. It has the property that if {\em either} precision or recall
is low, then the $f$-measure is decreased. The $f$-measure is useful
for dual assessments that include {\em both} precision and recall.
\begin{equation}\label{eq:f}
f_x=\frac{2 * prec_x * pd_x}{prec_x + pd_x}
\end{equation}
Given $n$ target classes {\em x,y,\ldots}, then one measure of the total
performance is $F$, i.e. the average $f$ measure seen in  all classes:
\begin{equation}\label{eq:F}
F=\frac{1}{n}\sum_i^nf_i
\end{equation} 
This measure
has the range $0\le F\le 1$. Note that  the {\em larger} the $F$ value, the {\em better}
the learned model for predicting all the target classes.

The $F$ measures were collected in
10-way  cross-validation
experiments. Such experiments assess a learned model using data {\em not}
used to generate it.
To conduct such an experiment,
the available date is divided into $N$ buckets.
For each bucket in a $N=10$-way cross-evaluation, a model is
learned on the nine of the buckets, then tested on the remaining
bucket.  
These studies
are the 
preferred evaluation method when the
goal  is to produce predictors intended to  predict future 
events~\cite{witten05}.

\begin{figure}
\scriptsize
\begin{alltt}
us >= 5                         => \_1 
am >= 3                         => \_5 
ex <= 2 AND rm <= 1 AND us >= 2 => \_5 
us <= 1 AND am >= 2             => \_5 
ex >= 2 AND am <= 1 AND us <= 2 => \_5 
ex <= 1 AND us >= 2             => \_4 
us >= 3 AND ex >= 3             => \_4 
ex <= 1                         => \_3 
us <= 2 AND ex <= 2             => \_3 
otherwise                       => \_2 
\end{alltt}
\caption{Rules from treatment 5e or \protect\fig{ff}. $F=0.95$. }\label{fig:5e}
\end{figure}
\begin{figure}
\scriptsize
\begin{alltt}
am >= 3 and us >= 3                         => \_5 
ex <= 2 and am >= 2 and us <= 1             => \_5 
am <= 1 and us >= 2 and us <= 2 and ex >= 2 => \_5
ex <= 1 and us >= 3 and am <= 1             => \_5 
ex <= 1 and us >= 2                         => \_4 
us >= 3 and ex <= 4 and ex >= 3             => \_4 
us <= 2 and ex <= 2                         => \_3 
otherwise                                   => \_1' 
\end{alltt}
\caption{SILAP2: the rules from treatment 4f of \protect\fig{ff}.
$F=0.95$. }\label{fig:4f}
\end{figure}

\subsection{Results}

\fig{ff} shows the $F$ values seen in 
10-way
{\em cross validation} experiments using RIPPER on 
on the four and five class systems.   
Several aspects of  \fig{ff} are noteworthy:
\bi
\item
The results of \fig{ff} are actually better than they may appear.
Cross validation 
reports the performance of a model that was learned using {\em less} than all the available data.
Hence, they
{\em under-estimate}
the performance of a learned models so the {\em actual} performance of our best learned models 
is {\em more than} the $F=0.95$ reported in \fig{ff}.
That is, data mining the SILAP data yielded near
perfect predictors for weighted issue severity
(i.e. nearly
perform at $F=1$). 

\item
No single feature was a best predictor for issue severity. All the experiments
with single features (L1,L2,us) performed worse than those that used
combinations of features. This explains a curious effect- single
features with poor correlations
to $W$ such as artifact maturity (am) (see \fig{corr}) proved to be quite useful.
\item
The high-ranked features listed in \fig{fss} 
performed better than using just the raw SILAP1 features; i.e. the WRAPPER
experiments were a success.
\item
Using just system name ($L1$)  performed poorly. This is good since
otherwise, 
there would be no generality in NASA's issue reports.
\item
The sub-system type ($L2$) proved to be a poor predictor of issue severity.
That is, there was no evidence from this data that any of
GNC, GROUND, CDH, or PAYLOAD are inherently more risky that any of the others.

\ei

The rules learned from \fig{fss}'s best treatments  are shown in \fig{5e} and \fig{4f}.
It is important to read these rules top to bottom since
a rule fires {\em only} if {\em all} the rules above it {\em fail}. 

The rules of \fig{5e} are 
adherent- the top rule says that use of standards (us) predicts for highest severity issues.
Recalling the above discussion, this aberration is due to quirks in the single project
with the highest severity errors. 

The rules of \fig{4f} combine the severity 1 and two issues into one class \_1'.
This avoids the problem of a small sample of severity one issues.
This rules seem saner than \fig{5e}. For example:
\bi
\item
The first rule
claims that mature products being extended using industrial standards
predicts for very low issue severity.
\item
The last rule predicts for high severity issues and that can be reached via poor use of
standards and poor developer experience.
\ei
Certain quirks with \fig{4f} can be explained with reference to the correlation matrix
shown above.
For example, proponents of the CMM model (e.g.~\cite{gibson06})
might raise a concern that these rules make no mention of CMM as a method of lowering 
defects.  However, \fig{corr} reports a strong correlation (67\%)
between CMM level (cl) and developer experience (ex). Given such a strong correlation,
a feature pruner such as WRAPPER could select either of cl {\em or}  ex for the final model.
That is, process maturity is tacit in \fig{4f} via the correlated concept of developer
experience. 


\begin{figure}[!t]
\scriptsize
\begin{center}
\begin{tabular}{r|r|r|r|r}
$p$=phase & ~task&  ~cost &~frequency&frequency*cost\\\hline
3. requirements

   &         3.5&         0.77&5&4\\
  &          3.4&         1.62&50&81\\
 &           3.3&         1.6&53&85\\
&            3.1&         1.46&90&131\\
&            3.2&         3.89&54&210\\

\cline{4-5}
 &              &             & sub-total&$\alpha_3=512$\\\hline
2. concept

&            2.5&         0.15&13&2\\
 &           2.6&         0.36&13&5\\
      &      2.4&         0.86&13&11\\
    &        2.2&         1.28&50&63\\
     &       2.3&         1.67& 54&90\\
&            2.1&         1.53&71&109\\

\cline{4-5}
 &              &             &  sub-total&$\alpha_2=281$\\\hline
         4.  design

      &      4.5&         1.43&5&7\\
       &     4.6&         0.63&35&22\\
    &        4.3&         2.50&9&23\\
   &         4.2&         4.31&31&134\\
& 4.1&         2.71&81&220\\
     &       4.4&         2.84&88&250\\

\cline{4-5}
 &              &             &  sub-total&$\alpha_4=655$\\\hline
5. implement

   &        5.11&         1.82&3&5\\
   &         5.5&         1.16&26&30\\
  &          5.4&         5.06&26&132\\
&            5.1&         3.49&81&283\\
        &    5.10&         3.8&81&313\\
 &           5.3&         10.28&53&545\\

\cline{4-5}
 &              &             & sub-total&$\alpha_5=1314$\\\hline

6. test

        &    6.4&         2.08&26&54\\
       &     6.3&        20.93&3&63\\
         &   6.5&         3.20&26&83\\
&            6.1&         2.30&88&202\\

\cline{4-5}
 &              &             & sub-total&$\alpha_6=402$\\\hline
\end{tabular}
\end{center}

\caption{Costs and frequency of \fig{silapModel} tasks.
Zero cells denote ``no data available''. }\label{fig:allocated}
\end{figure}

\begin{figure*}[!b]
\begin{center}
\scriptsize
\begin{tabular}{r@{~}l|r@{~}l|r@{~}l|r@{~}l|r@{~}l}
 &              & \multicolumn{8}{c}{$\frac{issues'}{cost*frequency} = \frac{\beta_p
\cdot  \Delta^{(6-p)}}{\alpha_p}$ }\\\cline{3-10}
&               & \multicolumn{4}{c|}{robotic} & \multicolumn{4}{c}{manned}\\\hline
$p =$&phase   &    $\Delta=2$ & &$\Delta=5$  & &$\Delta=2$ & &$\Delta=5$&\\\hline
2&concept     &    713 (18\%) & \rule{4mm}{2mm}  &  27,867 (39\%) & \rule{10mm}{2mm} & 39,216  (42\%) & \rule{10mm}{2mm}& 1,531,876  (66\%) & \rule{16mm}{2mm} \\
3&requirements&  2,688 (67\%) & \rule{17mm}{2mm} &  41,993 (58\%) & \rule{15mm}{2mm} & 51,067  (55\%) & \rule{13mm}{2mm}&   797,924  (34\%) & \rule{8mm}{2mm}  \\
4&design      &    286 ~(7\%) & \rule{2mm}{2mm}  &   1,788 ~(2\%) &                  &    828  ~(1\%) &                 &     5,178  ~(0\%) &                  \\
5&implement   &    165 ~(4\%) & \rule{1mm}{2mm}  &     412 ~(1\%) &                  &    950  ~(1\%) &                 &     2,376  ~(0\%) &                  \\
6&test        &    146 ~(4\%) & \rule{1mm}{2mm}  &     146 ~(0\%) &                  &    459  ~(0\%) &                 &       459  ~(0\%) &             \\\hline

 &    totals      & $\sum=3,998$   &                 & $\sum=72,206$    &  &   $\sum=92,521$ && $\sum=2,337,813$   &
\end{tabular}
\end{center}
\caption{Heuristic assessment of the relative merits of IV\&V tasks at different phases.
The values of $\alpha_p$ and $beta_p$ come from \fig{allocated}
and \fig{issues}. 
}\label{fig:phase}
\end{figure*}

\section{Task Pruning Heuristics}

SILAP2
can't distinguish between severity levels \_1 and \_2.
When these error potential predictors are applied to \fig{silapModel},
they 
can select slightly more tasks that what is strictly necessary:
in tasks 
2.4, 2.5, 2.6, 4.7, 5.6, 5.8, 5.10, 6.1 of
\fig{silapModel}, there are 8 cases
where
the tasks selected by $ep=5$ would not be selected when $ep=4$.
This is not an ideal situation
but, recalling the adherent rules of \fig{5e}, our current data does not support the generation
of good level \_1 predictors.

If budget permits, then all the tasks recommended
by SILAP2 should be applied. However, 
since SILAP2 selects {\em  more} than the minimal IV\&V tasks, the research
question becomes how to rank and, possibly, prune the recommended tasks.


%\begin{figure*}
%\scriptsize
%\begin{center}
%\begin{tabular}{r|r|r|r@{\%~}l|r|l}
%type & $\Delta$ & rank     & \multicolumn{2}{l|}{normalized 
%$\left(\frac{Issues\cdot 10^{(5-Severity)} \cdot \Delta^{(6-Phase)}}{Costs \cdot Frequency}\right)$ } &Task id & Task\\\hline
%robot&5&   1   &    52& \rule{52mm}{2mm}& 2.5& SW/User Requirements Allocation Analysis\\ 
% &&    2   &    22& \rule{22mm}{2mm}& 2.6&                    Traceability Analysis\\ 
%  &&   3   &     9&  \rule{9mm}{2mm}& 2.4&              Concept Document Evaluation\\ 
%   &&  4   &     4&  \rule{4mm}{2mm}& 3.4&                System Test Plan Analysis\\ 
% &&    5   &     4&  \rule{4mm}{2mm}& 3.3&        Interface Analysis - Requirements\\ 
%  &&   6   &     3&  \rule{3mm}{2mm}& 3.1&    Traceability Analysis -  Requirements\\ 
%&&     7   &     2&  \rule{2mm}{2mm}& 3.2&         Software Requirements Evaluation\\ 
%&&     8   &     2&  \rule{2mm}{2mm}& 2.2&         Software Architecture Assessment\\ 
%&&     9   &     1&  \rule{1mm}{2mm}& 4.6&                        Database Analysis\\ 
%&&    10   &     1&  \rule{1mm}{2mm}& 2.3&               System Requirements Review\\ 
%&&    11   &     1&  \rule{1mm}{2mm}& 2.1&                          Reuse Analysis*\\\hline 
%
%robot&2&     1   &    39& \rule{39mm}{2mm}& 2.5& SW/User Requirements Allocation Analysis\\
%&&     2   &    16& \rule{16mm}{2mm}& 2.6&                    Traceability Analysis\\ 
%&&     3   &     8&  \rule{8mm}{2mm}& 3.4&                System Test Plan Analysis\\ 
%&&     4   &     8&  \rule{8mm}{2mm}& 3.3&        Interface Analysis - Requirements\\ 
%&&     5   &     7&  \rule{7mm}{2mm}& 2.4&              Concept Document Evaluation\\ 
%&&     6   &     5&  \rule{5mm}{2mm}& 4.6&                        Database Analysis\\ 
%&&     7   &     5&  \rule{5mm}{2mm}& 3.1&    Traceability Analysis -  Requirements\\ 
%&&     8   &     3&  \rule{3mm}{2mm}& 3.2&         Software Requirements Evaluation\\ 
%&&     9   &     2&  \rule{2mm}{2mm}& 5.4&                System Test Case Analysis\\ 
%&&    10   &     1&  \rule{1mm}{2mm}& 6.4&             System Test Results Analysis\\ 
%&&    11   &     1&  \rule{1mm}{2mm}& 5.5&               Software FQT Case Analysis\\ 
%&&    12   &     1&  \rule{1mm}{2mm}& 4.2&               Software Design Evaluation\\ 
%&&    13   &     1&  \rule{1mm}{2mm}& 2.3&               System Requirements Review\\ 
%&&    14   &     1&  \rule{1mm}{2mm}& 2.2&         Software Architecture Assessment\\ 
%&&    15   &     1&  \rule{1mm}{2mm}& 2.1&                          Reuse Analysis*\\ \hline
%
%human&5&     1   &    59& \rule{59mm}{2mm}& 2.5& SW/User Requirements Allocation Analysis\\ 
%&&     2   &    24& \rule{24mm}{2mm}& 2.6&                    Traceability Analysis\\ 
%&&     3   &    10& \rule{10mm}{2mm}& 2.4&              Concept Document Evaluation\\ 
%&&     4   &     2&  \rule{2mm}{2mm}& 2.2&         Software Architecture Assessment\\ 
%&&     5   &     1&  \rule{1mm}{2mm}& 3.4&                System Test Plan Analysis\\ 
%&&     6   &     1&  \rule{1mm}{2mm}& 3.3&        Interface Analysis - Requirements\\ 
%&&     7   &     1&  \rule{1mm}{2mm}& 2.3&               System Requirements Review\\ 
%&&     8   &     1&  \rule{1mm}{2mm}& 2.1&                          Reuse Analysis*\\\hline 
%
%
%human&2&     1   &    56& \rule{56mm}{2mm}& 2.5& SW/User Requirements Allocation Analysis\\ 
%&&     2   &    23& \rule{23mm}{2mm}& 2.6&                    Traceability Analysis\\ 
%&&     3   &    10& \rule{10mm}{2mm}& 2.4&              Concept Document Evaluation\\ 
%&&     4   &     2&  \rule{2mm}{2mm}& 3.4&                System Test Plan Analysis\\ 
%&&     5   &     2&  \rule{2mm}{2mm}& 3.3&        Interface Analysis - Requirements\\ 
%&&     6   &     2&  \rule{2mm}{2mm}& 2.2&         Software Architecture Assessment\\ 
%&&     7   &     1&  \rule{1mm}{2mm}& 5.4&                System Test Case Analysis\\ 
%&&     8   &     1&  \rule{1mm}{2mm}& 4.6&                        Database Analysis\\ 
%&&     9   &     1&  \rule{1mm}{2mm}& 3.2&         Software Requirements Evaluation\\ 
% &&   10   &     1&  \rule{1mm}{2mm}& 3.1&    Traceability Analysis -  Requirements\\ 
% &&   11   &     1&  \rule{1mm}{2mm}& 2.3&               System Requirements Review\\ 
%&&    12   &     1&  \rule{1mm}{2mm}& 2.1&                          Reuse Analysis* 
%\end{tabular}
%\end{center}
%\caption{Value of \fig{silapModel} tasks; values normalized to sum to 100.}\label{fig:value}
%\end{figure*}

\fig{allocated} shows the available $cost$ per task and the  $frequency$ of performing tasks
 on the current
IV\&V projects.  
For proprietary reasons,
these values are normalized to sum to 100.  The value $\alpha_p$ sums total $cost*frequency$ for each phase $p$.

\fig{issues} showed the issues recently found by IV\&V
in 9 robotic missions and 7 man-rated missions. 
In that figure,
the value $\beta_p$ sums the issues found in phase $p$ times a factor that weights high severity
issues exponentially larger than low severity issues.  

An ideal IV\&V tasks finds many issues, early in the life cycle, at low cost, after
performing just a few tasks; i.e. it maximizes \begin{equation}\label{eq:bg}
\frac{issues\; found\; early}{effort\; expended}
=\frac{\beta_p
\cdot  \Delta^{(6-p)}}{\alpha_p} 
\end{equation}
In this expression, the $\Delta$ term rewards issues found earlier in the life cycle. 

\fig{phase} is a heuristic assessment of 
performing IV\&V tasks at different phases of the life cycle. 
The four results
columns of that figure compute \eq{bg}
for
$\Delta\in\{2,5\}$ (see \fig{dabney06a}) and for the robotics and man-rated mission data of \fig{issues}.
Each column shows the \eq{bg} values as a percentage of the totals for that column.

From \fig{phase}, we conclude that
if budgets do not permit performing all the tasks recommended by SILAP2, then late life cycle
tasks should be pruned away {\em before} concept and requirements phased tasks. Overwhelming,
most of the cost-benefit of IV\&V is seen in the these first two phases.

These conclusions from
\fig{phase} are not without precedent. 
Writing over a decade ago, Higuera and Haimes~\cite{hig96}
reviewed classes of risk associated
with the Software Engineering Institute's risk taxonomy. They concluded
that the risks associated with product engineering, development environments,
and the general development program were roughly evenly distributes (30\%, 33\%, 37\%).
However, \fig{huguera} shows the 
risk within  product engineering (i.e. the region usually studied by IV\&V).
Note that 
their risk distribution matches the general pattern of \fig{phase}: late
life cycle activities offer very little to the overall risk.




Returning now to the eight problem cases of SILAP2,
we note
that 5 of those cases are late cycle tasks; i.e. can be pruned without degrading much of the value
of IV\&V. 
As to the remaining 3 problem cases (the concept phase tasks of 2.4, 2.5, 2.6),
\fig{allcosts} shows that these three are, relatively speaking,
the least costly of all the IV\&V tasks. That is, even if they are selected inappropriately,
they will add little to the total IV\&V cost. 
Hence we assert that the known problems with SILAP2
are negligible. 

\begin{figure}
\begin{center}
\scriptsize
\begin{tabular}{r@{~}r@{~}l}
requirements & 53\% & \rule{53mm}{2mm}\\
design & 27\% & \rule{27mm}{2mm}\\
integration \& test & 14\% & \rule{14mm}{2mm}\\
engineering specialties & 4\% & \rule{4mm}{2mm}\\
coding \& unit test & 2\% & \rule{2mm}{2mm}\\
\end{tabular}
\end{center}
\caption{Patterns of risk within product engineering; from \protect\cite{hig96}.}\label{fig:huguera}.
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[width=2in]{allcosts.pdf}
\end{center}
\caption{All task costs from \fig{allocated}, sorted.}\label{fig:allcosts}
\end{figure}
\section{Conclusion}

\subsection{NASA-specific Conclusions }

\subsubsection{Revise SILAP1}

SILAP1 needs some changes.
\fig{epcow} shows that
SILAP1's calculated consequence and error potential values are poor predictors for
issue severity. 

\subsubsection{Adopt SILAP2}

While the derived features of SILAP1 should be removed,
the raw features of SILAP1 are demonstrably useful. As seen in \fig{ff},
when processed by the WRAPPER
and RIPPER, these raw features can be combined into an issue severity
predictor was learned with a near perfect $F$ measure.

Our proposed SILAP2 model retains (a)~the task selection table of \fig{silapModel};
(b)~the raw SILAP1 features of \fig{silapFeatures}; and (c)~the consequence of error (co)
calculations of \fig{SILAP.awk}. However, it replaces the error potential calculation
with the rules of \fig{4f}.

The following
syntactic redefinition is required to map the output of the rules to
the task selection table:
\bi
\item
Error potentials of 1,2,3,4,5 (in \fig{silapModel}) correspond to issue severity predictions
(from \fig{ff}) of $\_5,\_4,\_3,\_1',\_1'$, respectively.
\ei


\subsubsection{Revise Current IV\&V Task Assignments}

Since SILAP1 underestimates project risks, and SILAP2 is exhibiting good performance,
all current IV\&V task assignments should be reassessed using SILAP2.


\subsubsection{Revise NASA's Data Collection Procedures}

The hundreds of  records found by this study were useful and informative. However, for a national
software development organization, the goal should be thousands, or tens of thousands, of records. 
Also, the performance of the SILAP1 raw features in SILAP2 
suggests that when NASA IV\&V next
changes 
its databases it should, at the very least, retain the raw SILAP1 features.

\subsection{General Conclusions}

\subsubsection{Coping with Real World Data}

Ideally, organizations are stable enough to develop a central reference model for their
software development. Such a reference model is very useful for defining and interpreting
data collection  and maturing a ``lessons learned'' knowledge base. Such an ideal situation existed
with the  NASA SEL from 1983 to 1995. 

Given the current pace of change of NASA 
it is unrealistic to deploy the learning organization framework within NASA. Nevertheless,
as shown here, heuristically gathered data can still be used to generate a knowledge base
with a surprisingly high effectiveness ($F=0.95$ in a 10-way cross-validation).

We attribute the high performance of our learners to the methods used
to remove the noise present in most real world SE data sets. 
To remove noisy data, it is good to apply operators that prune away superfluous parts of the data
and learned model. This study recommends two such operators: the back-select techniques of RIPPER and the 
WRAPPER feature subset selector. 


\subsubsection{Define Research Targets}

Many techniques discussed widely in the SE literature (e.g. formal methods)
have been rejected by this analysis as not cost-effective. It is important to stress
that our conclusions are {\em only} for IV\&V. Embedded V\&V is performed in a different
business context and, in that context, using (e.g.) model checkers to check for temporal logic
violations in source code might be particularly cost-effective.
The lesson here is that theoretical SE research needs to understand more about industrial SE practice.
For example, proponents of a SE method should be aware of the business contexts
where their method are applicable or impractical. Ideally, the literature should
routinely contain research targets 
such as 
\begin{quote}
``when generating  formal models from source code costs {\em less than} (say) 
10\% of the source code generation,
then formal methods will become practical for (one of) \{agile development, embedded V\&V, etc\}''.
\end{quote}
This research target should be justified using industrial data. We would argue that funding bodies
should identify such research targets, then selectively fund research according to how well
a research proposal might contribute to those targets.


\subsubsection{When to Perform IV\&V}

 Early life-cycle IV\&V tasks (in concept and requirements phase)
are far more effective than IV\&V conducted later in the life cycle.
 This result
has been reported before (e.g.~\cite{boehm88}). Usually, the benefits were shown as an exponential {\em curve}. 
In contrast, 
we found more a {\em cliff} than a curve.
Compared to early life cycle IV\&V, the merits of late life cycle IV\&V are 
very meager. In \fig{phase},  performing IV\&V in the design, implementation, and testing phases
was between 0 to 15\% of the total added value. Due to the complexities of collecting the data for
that figure,
those percentages have a degree of uncertainty. However, we would defend the general pattern of
\fig{phase}: overwhelmingly, early life cycle IV\&V is far more valuable than later life cycle.


\subsubsection{Collect $\Delta$ Data}

\fig{phase} showed that the relative merits of concept-phase vs requirements-phase IV\&V is crucially dependent on the
cost-to-fix
$\Delta$ value. At $\Delta=2$, requirements IV\&V is more valuable than concepts IV\&V (for both
robotic and manned missions)
but
at $\Delta=5$, the pattern reverses and concept IV\&V is more valuable (particularly for manned missions). 
Organizations should routinely
collect data like \fig{dabney06a} in order to find their own most cost-effective
 phase to perform IV\&V.

\subsubsection{IV\&V Task Selection}

If budgets are tight, we advise skipping late life cycle activity and focus more on the cheaper
early life cycle activities. In this context, five tasks are worthy of special attention.

According to \fig{allocated},
the cheapest and earliest life cycle activities are 
concept document evaluation (2.4), sw/user requirements allocation analysis (2.5),
and traceability analysis (2.6).
Since these tasks are so cheap, and offer the most benefit earliest in the life cycle,
we would recommend that they are {\em always} 
performed.

Also, as shown in \fig{allcosts},
two tasks have an outstandingly
large cost: interface analysis, code (5.3); and simulation analysis (6.3). Based
on the above, we would argue that these most expensive, late life cycle, least cost-effective
tasks should only be performed if there is a very pressing business case to do so.

\bibliographystyle{IEEEtran}
\bibliography{refs}
\input{tim/bio} 

\input{markland/bio} 


\input{ken/bio} 
\input{christina/bio} 
\input{melissa/bio} 
\input{julian/bio} 
\end{document}

fss ok to throw away much of the data.

the world is cross-correlated.
things that look poor in isolation work well in isolation caution against the single spreadsheet
plot where one feature is plotted against the target variable

se complex, but not some complex that we can't find patterns

se characterizes but much theory and very little experimental verification. 

