The is much formal and informal evidence for the value of IV\&V.
By ``formal'' evidence, we mean that evidence in published sources
that  can be verified by another researcher.
Based on our interviews with NASA IV\&V personnel, we offer the informal evidence
of \fig{informal}.
\begin{figure}[!t]
\footnotesize
\begin{tabular}{|p{0.95\linewidth}|}\hline
\bi
\item
``Even when developers assess their own products, they may be focused
on a very small part of a much larger system. IV\&V workers, on the other hand,
can explore larger parts of the system.''
\item
``When faced with schedule pressure,
developers can race to finish a product without taking adequate time
for self-review of that work. These reviews can be conducted by
IV\&V.''
\item
``Historically, it has been observed at NASA
that development teams can change personnel faster than the IV\&V team.
Hence, the IV\&V team can come to understand the software as well or better
than the development team. 
For example, in several NASA mulit-year projects, most of the issues
were found
by the IV\&V team, and not the developers.
 Also, in numerous cases, the IV\&V team can recall  why (e.g.)
three years ago, some crucial
decision
was made.
That is, IV\&V has become NASA's corporate memory for software-related
issues.''
\item
``The technical skills of NASA's IV\&V's team are acknowledge by
(some) of NASA's contractors.
In one case, the prime contractor on a NASA project
found funds within its own budget to pay for more IV\&V,
over and above the amount required by their NASA contract.''
\ei
\\\hline
\end{tabular}
\caption{Informal evidence for the benefits of IV\&V, taken
from interviews
with NASA IV\&V personnel, January to March 2007.}\label{fig:informal}
\end{figure}
\begin{figure}
\begin{center}
{\scriptsize 
\begin{tabular}{r@{~}r|r@{~}r@{~}r@{~}r@{~}r@{~}r}
\multicolumn{2}{c}{~}&\multicolumn{6}{c}{Phase issue found}\\\cline{3-8}
&Phase issue        & f=1        & f=2   & f=3  & f=4 &f=5 &f=6\\
i&introduced   &Requirements& Design& Code &Test &Int& Operations\\\hline 
1&Requirements &           1& 5     & 10   & 50  & 130& 368\\
2&Design        &            &  1   & 2    & 10  &  26&  74 \\
3&Code          &            &      & 1    & 5   & 13 &  37\\ 
4&Test           &           &      &      & 1   & 3 & 7\\ 
5&Integration    &          &       &      &     & 1 & 3\\\hline
   &$\Delta$ = mean $\left(\frac{C[f,i]}{C[f,i-1]}\right)$
                   &&5          & 2    & 5   &2.7     &2.8\\            
\end{tabular} }
\end{center}
\caption{$C[f,i]$: The cost-to-fix escalation factors, relative to
fixing an issue in the phase where it was found ($f$) versus
in the phase where it was introduced ($i$).
Last row shows the cost-to-fix delta if the issue  introdced in phase $i$ 
is fixed immediately afterwards in phase $f=i+1$. From~\cite{dabney06a}.}
\label{fig:dabney06a}
\end{figure}
\begin{figure}
\begin{center}
{\footnotesize
\begin{tabular}{r@{~}|r@{~}l@{~}|r@{~}l}
phase & \multicolumn{2}{c|}{IV\&V}&\multicolumn{2}{c}{no IV\&V}\\\hline
requirements& 16&\rule{8mm}{1.5mm}& 0&\\
high-level design& 20&\rule{10mm}{1.5mm}& 2 & \rule{1mm}{1.5mm}\\
low-level design& 31&\rule{15mm}{1.5mm}& 8&\rule{4mm}{1.5mm}\\
coding \& user testing& 24&\rule{12mm}{1.5mm}& 34&\rule{17mm}{1.5mm}\\
integration \& testing& 6&\rule{3mm}{1.5mm} & 14 & \rule{7mm}{1.5mm}\\\hline
totals& 97&& 58 & 
\end{tabular} }
\end{center}
\caption{The Arthur et.al. controlled experiment comparing defects found
with and without IV\&V. From \protect\cite{arthur99}.}\label{fig:arthur}
\end{figure}

Formal evidence for the benefit of IV\&V comes from the software engineering
literature.
Zelkowitz and Rus discuss the Space Shuttle IV\&V  program in the 1990s~\cite{zelkowitz01}. 
They report a large decrease in major flight defects 
once the IV\&V program was introduced, and   that IV\&V uncovered and resolved
certain
shortcomings in the shuttle's testing program. 
Their most significant finding was that almost half the issues found by
IV\&V were found very early in lifecycle of shuttle software upgrades.

Finding errors early in the lifecyle is critical to software cost reduction.
In 1988, Boehm \& Papaccio 
cautioned that fixing or reworking software is far
cheaper earlier in the lifecycle than later ``by factors
of 50 to 200''~\cite{boehm88}.
This effect has been widely documented in other research:
\bi
\item
\fig{dabney06a} shows 
the cost of quickly fix an issue relative to
leaving it for a later phase (data from four NASA projects~\cite{dabney06a}). 
The last line of that table shows that
delaying issue resolution 
even by one  phase
increases the cost-to-fix of $\Delta=2 \ldots 5$.
\item
A panel at IEEE Metrics 2002~\cite{shu02}  debated the general benefit
of early lifecycle defect removal. They concluded that
finding and fixing  severe software problems
after delivery is often 100 times more 
expensive than finding and fixing it during 
the requirements and design phase. They panel also commented
that for 
non-severe software 
defects, that factor can be much lower down to a factor of two.
Note that, in either case, it is cheaper to fix earlier than later.
\item
Wallace \& Fugii~\cite{wallace89} discuss 
a 
study where IV\&V was started at different phases.
Starting  at the coding phase
only saved 20\% of the total V\&V costs (embedded + indepedent).
However,
starting earlier at the requirements
phase saved 92\% to 180\% of the toatl V\&V costs.
\item
Arthur et.al.~\cite{arthur99} conducted a small controlled experiment
where
a dozen engineers at NASA's Langley Reseaerch Center were 
split into development and IV\&V teams. The same application
was written with and without IV\&V. 
\fig{arthur} shows the results:
(a)~more issues were found using IV\&V than without; (b)~the issues
were found much
earlier.
This is a compelling argument for IV\&V. 
Even if the IV\&V team found the same bugs as the development
team, but found them earlier, the cost-to-fix for the project
would be reduced by a significant factor.
\item
Using \fig{dabney06a},
Dabney et.al.~\cite{dabney06a} caluclated that 
every dollar
spent on IV\&V 
returned to NASA \$1.21, \$1.59, \$5.53, and \$10.10 (on four NASA projects). 
\ei

\input{tim/silapModel}



\section{IV\&V: How?}

\PARstart{T}{he} evidence shown above makes a clear case for the value of 
IV\&V. 
Corporations get most benefit from
IV\&V if they do  {\em not} viewed it as an end-lifecycle
activity that focuses on source code. Rather, IV\&V is should be a full lifecycle activity
devoted to software assurance and greatly reducing the cost of reworking software.
 
Having argued for full lifecycle IV\&V, the next question an manager will ask is
``how to best structure a particular IV\&V project''?  This is a difficult question
to answer:
\bi
\item
 Several researchers offer detailed taxonomy of standard NASA 
faults~\cite{hayes05,malin07},
but
do not address what methods are most cost-effective at finding the faults. 
\item
Lutz reports
studies on the most frequent kind of faults seen in deep space NASA missions but does
fully address the issue of best methods for fault removal~\cite{lutz04}. 
\item
Leveson talks in general terms
about how increase the safety of software~\cite{leven95}
 but since her preferred methods would replace
current IV\&V tasks~\cite{levenson96}, it is difficult to map that advice into
IV\&V task selection.
\ei
Various authorities list nearly 100 V\&V and IV\&V methods:
\bi
\item
Wallace \& Fugii offer 20 essential and 36 optional
tasks for software V\&V~\cite{wallace89}. 
\item
Table 2 of the  IEEE-1012 standard for software V\&V
offers a list of 52 minimal tasks for IV\&V. Each task is selected by a combination of development phase and
estimated severity in project errors~\cite{1012}.  
\item
NASA's local variant of IEEE-1012 lists the 40 pre-deployment
tasks shown in \fig{silapModel}. 
Each task is selected by an estimate of project {\em error potential} (a.k.a. liklihood)
and {\em consequence} (a.k.a. risk). 
These estimates come from the SILAP model, discussed later in this article.
\ei
If budget permits, all these tasks should be peformed.
However, in the usual case of limited budgets, software engineers must carefully
select their IV\&V tasks.
Quite possibly, some of those tasks are cheapet to apply while tending to find more issues.
But which?

In the literature, there are very few attempts to assess the relative
cost-effectivenss of IV\&V techniques.
Most of the reports
take the form of (e.g.):
\bi
\item
Easterbrook et.al.~\cite{easter98} who discuss the merits of using a single
lightweight formal method for IV\&V 
requirements models on one NASA project;
\item Hayes, Dekhtyar, and Sundraram who explore
the relative merits of
different tools for finding links between requirements documents~\cite{hayes06}. That analysis includes a comparison
of human expert and state-of-the-art commercial tools and a new research prototype.
\item
For more examples of singleton evauluations of one method on one project,
see
the proceedings of the annual
sympoisum and repository of NASA's Software Assurance Research Program\footnote{\url{http://sas.ivv.nasa.gov/conclusions/index.php} and
\url{http://sarpresults.ivv.nasa.gov}}.
\ei

In a limited number of publications, some attempt is made to generalize over
multiple projects, multiple methods, or projects extending over many years.
Those publications
include:
\bi
\item Dabney et.al.'s study, discussed above.
\item Menzies et.al. discussion of the merits of different data miners to learn predictors for software defects~\cite{me03q,me07b}.
\item Raffo et.al.'s software process of a standard waterfall model, modified to handle IV\&V tasks~\cite{raffo04}.
\ei

However, even taken in combination, these publications cover just a small
fraction of the tasks shown in \fig{silapModel}. 
Hence, in the sequel, we will use a screening study of NASA's IV\&V issue reports
to rank IV\&V tasks according to their relative cost-benefits.


%A random
%sample of these were read including all the journal-level papers
%plus all the papers recommneded by researchers in the field.
%Many of the papers
%read in detail, including all the seven journal-level papers,
%The random sample made extensive reference to the SEL papers
%described above, plus an earlier Boehm \& Papaccio paper
%from 1988~\cite{boehm88}. In one case, a forward reference
%was found from a journal paper to a subsequent conference panel
%that debated the conclusions of that paper. 
%
%Other literature was not so forthcoming about results relating
%to IV\&V and the specifics of defect reduction. For example:
%\bi
%\item
%A standard text on software metrics spends more effort
%on how to
%define a measurement program than 
%on specific results~\cite{fenton96}. 
%\item
%A standard text on IV\&V did not ground its recommendations in
%an experienced-based manner~\cite{lewis90}.
%\ei
%In all, seven journal-level papers
%were found and this study focused on those, plus an earlier
%Boehm prominent papers
%
%To this set, sever
%All hardware-only papers papers were rejected from the
%corpus.
%Only seven journal-level papers appered in that corpus
%and, of those,  only t
%journal-level papers in the that cThis study
%focused on the 7 journal-level papers in that corpus.
% papers range from 1989 to 2006.
%For this study, all journal-level and
%
