\PARstart{E}{vidence}-based reasoning is becoming common in many fields. It is
widely enshrined in the practice and teaching of medicine, law,
and management, for example. Evidence-based approaches demand
that, among other things, that
 practitioners systematically:
\bi
\item
track down
the best evidence relating to some practice; 
\item
critically appraise
that evidence for validity, impact, and applicability; 
\item
and carefully
document it.  
\ei

One proponent of evidence-based software engineering
is David Budgen of Durham University. In the Internet age, he
argues, many sources of supposed knowledge
(Google, Wikipedia, digg.com,
and so on)
surround us.  He poses\footnote{
Keynote address, CSEET'06. 
} the following challenge: 
``how
should we train students to assess all that information and to
separate the sense from the nonsense?''. In his view, before we can
denounce some inaccuracy in, say, Wikipedia, we must first look to
our own work and audit our own results.  

This article is an evidence-based analysis
of independent
software
verification and validation (IV\&V). 
We are concerned with what we can {\em show}, not just
{\em claim}, about IV\&V.

The time is right for this article.
In their 1989 article  
{\em Software Verification
and Validation (V\&V): An Overview}~\cite{wallace89}, 
Wallace \& Fugii  
distinguished between
\bi
\item {\em Embedded V\&V}, which  is done in-house by the developers or the users; 
\item
And {\em independent V\&V}  (IV\&V)
which is done externally by a group that might choose to explore very different
issues to the embedded V\&V team\footnote{Annex C of the IEEE 1012-2004~\cite{1012} 
standard
offers a more detailed definition of IV\&V. That definition retains Wallace \& Fugii's essential 
disctinction between internal (embedded) and external (independent) V\&V.}.
\ei


Since 1989,  many organizations have implemented IV\&V
programs including space agencies in Europe, Japan, and American.
As shown in \fig{hist}, NASA's software IV\&V program is one of the
largest: each year, dozens of projects are checked by
an external team.
As far as we know, no other
organization has insight into such a broad range of software projects.
This affords NASA an unequalled opportunity to identify the most cost-effective
software IV\&V tasks.

Based on NASA's experience,
two issues will be explored:
\bi
\item
{\em Indentifying projects that need more IV\&V:}
What are the minimum set of project characteristics that most predicts for
a problem software project?
\item
{\em Conducting IV\&V in a cost-effective manner:}
What are the minimum set of cost-effective IV\&V tasks that most finds specific issues
in software?
\ei
We show that it is possible to build
effective predictors for projects that
need more IV\&V.
{\em Screening experiments} will be conducted,
that reduce a larger set of factors
to just those with
an
important effect on the severity levels of issues found by IV\&V.
The simplicity of the predictors will be surprising. Several factors
that the literature suggests {\em should} predict for
greater defects are absent from our learned predictors.

We aslo show that
the list of most cost-effective IV\&V tasks
can be readily found.
This list will be quite small.
suggesting that many of the  current lists of proposed
IV\&V tasks (e.g.~\cite{1012,costello05,wallace89}) are
overly elaborated.

Ths specific conclusions of this study are two-fold.
\be
\item
NASA has 
has been consistently {\em under-estimating}
the risks associated with NASA software projects. 
\item
IV\&V is often applied to static code artifacts.  
We show here the
analysis of such late lifecycle artifacts is one of the least
cost-effective IV\&V methods.  
The {\em effectiveness} of early lifecycle  V\&V
has been made before (e.g.\cite{boehm88,dabney06a,shu02,wallace89,arthur99}).
However, to the best of our knowledge,  this is the first clear result relating to
the relative {\em ineffectiveness} of later lifecycle V\&V.
\ee

\noindent
There are two business-level implications of the above:

\bi
\item
NASA must expand its IV\&V program.
\item
All organisations, not just NASA, should
{\em reduce} late lifecycle vertification-based
IV\&V (that focuses on code artificats) and {\em increase}
early life cycle validation-based IV\&V (that focuses
on the business context of a project).
\ei

\noindent
The rest of this paper is structured as follows.
XXX. 
%Our results will be contrasted against certain promiment studies into effective
%defect indentification/removal. 
%There will be some surprises.
%For example,
%predicting problematic software projects is surprisingly simple.
%In data mining experiments with 15 project descriptions,
%  this study found that three factors were enough to predict for problem projects,
%with a very high degree of precision.
%
%Also, we can audit the various lists of proposed
%IV\&V tasks:
%\bi
%\item
%Wallace \& Fugii offer 20 essential and 36 optional
%tasks for software V\&V. 
%\item
%The IEEE-1012 standard for software V\&V~\cite{1012}
%lists over 60 tasks for V\&V prior to deploying software.
%\item
%NASA's local variant of IEEE-1012 lists the 40 pre-deployment
%tasks shown in \fig{silapModel}.
%\ei
%Using the NASA data, we can find 
%a small subset of these that are outstandingly cost-effective.

%Many large corporations have similar software
%V\&V activities. However, it is not always in their best
%interest to release data or results to the public. 
%For example, the data used in this study
%is freely available\footnote{See \url{promisedata.org/repository}.}


%.
%
%plitprojects*years= 
%($IV\&V$) is one of the largest. \fig{hist} shows the number of NASA development
%projects that receive IV\&V 
% 
% 
% The Japanese    
%Sadly, little of the software engineering literature would satisfy
%a proponent of evidence-based software engineering (but see some
%exceptional counter examples such as~\cite{jorg04,jorgensen04}).
%For example, what formally documented evidence exists that the
%current set of software verification and validation preactices  are
%{\em best practices}?  For example, the IEEE 1012-1998 standard for
%Software Verification and Validation lists nearly a 100 tasks they
%recommend and best practice for V\&V. However, despite 50 years of
%software development, we know of no definitive evidence-based source
%describing which of those tasks are least cost-effective.
%
\begin{figure*}
\footnotesize
\begin{center}
\begin{tabular}{r|r@{~}l|p{2.75in}|l}
year & & \#IV\&V project at Fairmont IV\&V &   notes & oversight\\\hline
1988 &n/a & &Space shuttle begins IV\&V&\\
1991 &n/a& &WVU awarded grant to build Farimont IV\&V& NASA headquarters (east  coast)\\\hline
1994 &1  &\rule{1mm}{1.5mm} &International space station IV\&V at Fairmont begins &NASA AMES (west coast)\\
1995& 1 &\rule{1mm}{1.5mm}&&\\
1996& 2 &\rule{2mm}{1.5mm}&&\\
1996& 3 &\rule{3mm}{1.5mm}&New contracts enables IV\&V for all NASA projects
        (but projects must fund IV\&V from their own line items) &\\
1997& 3 &\rule{3mm}{1.5mm}&&\\
1998& 3 &\rule{3mm}{1.5mm}&&\\
1999& 12 &\rule{12mm}{1.5mm}& NASA mandates IV\&V for all NASA software&\\\hline
2000& 15&\rule{15mm}{1.5mm}& Facility oversight moves to Goddard Space Flight Center& GSFC (east coast)\\
2001& 20 &\rule{20mm}{1.5mm}&&\\
2002& 36&\rule{36mm}{1.5mm}&&\\
2003& 42 &\rule{42mm}{1.5mm}&IV\&V now funded from a central agency source. &\\
2004& 37&\rule{27mm}{1.5mm}&&\\
2005& 24&\rule{24mm}{1.5mm}&SILAP data collection begins&\\
2006& 26&\rule{26mm}{1.5mm}&&\\
2007& 24&\rule{24mm}{1.5mm}&&\\
\end{tabular}
\end{center}
\caption{History of NASA IV\&V.}\label{fig:hist}
\end{figure*}
%To address this problem, we turned to the historical records of
%NASA's software independent verification and Validation facility
%(IV\&V) facilty.
%The facility captures its findings in its
%Project and Issue Tracking System (PITS)
%database.  
%The data in PITS has been collected for more
%than 10 years and includes issues on robotic satellite missions and
%human-rated systems.  
%To the best of our knowledge, PITS is our
%best current source 
%for identifying the most cost-effective tasks within  the IEEE 1012-1998
%standard. 
%We can know of no bigger database, that we can access, that stores
%more deta from more projects\footnote{statistics- size of PITS}.
%
%This report records our findings from PITS. In summary, PITs suffers
%

%Hence, the specfic contribution of this paper is  as follows:
%\be
%\item The documentation of a large scale industrial V\&V process;
%\item An audit of that process;
%\item Specific recommendations on how to:
%\be
%\item Identify problematic projects that need more V\&V;
%\item Select the most cost-effective V\&V tasks;
%\ee
%\item The identification of future directions for improving V\&V.
%\ee



