%\documentclass{article}
\documentclass[journal,compsoc]{IEEEtran}
\usepackage{color,ifthen}
\newcommand{\aive}{{\"aive}}
\newcommand{\sq}[2]{%
    \definecolor{thiscol}{gray}{.#2}%
        \ifthenelse{#2<50}%
            {\color{white}}%
            {\color{black}}%
        \colorbox{thiscol}{\makebox[2em]{#1}}}


\usepackage{wrapfig}
\usepackage{pifont}
\usepackage{times}
\usepackage{alltt}
\usepackage{graphicx}
\usepackage{url}
\usepackage{eso-pic}
\usepackage{color}
\usepackage{type1cm}
\makeatletter
  \AddToShipoutPicture{%
    \setlength{\@tempdimb}{.5\paperwidth}%
    \setlength{\@tempdimc}{.5\paperheight}%
    \setlength{\unitlength}{1pt}%
      \makebox(600,900){\rotatebox{45}{\textcolor[gray]{0.80}{\fontsize{9cm}{9cm}\selectfont{Draft5}}}}}
\makeatother
%
% IEEE Computer Society needs nocompress option
% requires cite.sty v4.0 or later (November 2003)
%\usepackage[nocompress]{cite}
% normal IEEE
 \usepackage{cite}
 \usepackage{stfloats}

% correct bad hyphenation here

\usepackage{tim/tricks}
\usepackage{ken/tricks}
\usepackage{melissa/tricks}
\usepackage{markland/tricks}
\usepackage{butch/tricks}

\begin{document}

\title{Learning Better IV\&V Practices}
%
\author{Tim~Menzies~\IEEEmembership{Member,~IEEE}%
\thanks{
Dr. Menzies is with the Lane Department of Computer Science and Electrical Engineering,
West Virginia University, USA; \protect\url{tim@menzies.us}},
Markland Benson~\IEEEmembership{Member,~IEEE}% 
\thanks{Mr. Benson is with NASA, formerly at NASA's
IV\&V facility, Fairmont, West Virginia \protect\url{Markland.J.Benson@nasa.gov}},\\
Ken Costello%
\thanks{Mr. Costello is with NASA IV\&V: \protect\url{Kenneth.A.Costello@nasa.gov}},
Christina Moats% 
\thanks{Ms. Moats is with NASA IV\&V: \protect\url{Christina.D.Moats@nasa.gov}},
Melissa Northey% 
\thanks{Ms. Northey is with NASA IV\&V: \protect\url{Melissa.S.Northey@nasa.gov}},
and
Julian Richardson%
\thanks{Dr. Richardson is with RIACS at NASA Ames \protect\url{julianr@email.arc.nasa.gov}}
\thanks{
This research was conducted at
West Virginia University 
under NASA sub-contract project 100005549, task 5e, award 1002193r.
Reference herein to
any specific commercial product, process, or service by trade name,
trademark, manufacturer, or otherwise, does not constitute or imply
its endorsement by the United States 
Government.
}
\thanks{
See \protect\url{http://menzies.us/pdf/07ivv.pdf} for an earlier draft.}
}

\markboth{Journal of ???,~Vol.~X, No.~Y, April~2007}%
{Menzies  \MakeLowercase{\textit{et al.}}: 
Learning Better IV\&V Practices}

\maketitle

%------------------------------------------------------------------------- 
\begin{abstract}
After data mining NASA IV\&V data, we offer (a) an
   early life cycle predictor for project issue frequency and
   severity; (b) an IV\&V task selector (that used the predictor to
   find the appropriate IV\&V tasks); and (c) pruning heuristics
   describing what tasks to ignore, if the budget cannot
   accommodate all selected tasks. In 10-way cross-validation
   experiments, the predictor performs very well indeed: the average
   f-measures for predicting four classes of issue severity was
   over 0.9.

This predictor is built using public domain data and software.
To the best of our knowledge, this is the first reproducible
report of a demonstrably useful predictor for issue frequency and severity
that can be applied early in the life cycle.
\end{abstract}
\section{Introduction}
\PARstart{A}{ll} software has defect issues,  some of which are seen by developers.
We  seek
a predictor for the {\em number}
and {\em severity} of issues that a project will produce.  If 
applied {\em early} in the life cycle then
such a  predictor could check if
extra funding is required for project verification and validation (V\&V).

Wallace and Fujii~\cite{wallace89} distinguish:
\bi
\item {\em Embedded V\&V} is performed in-house by developers; 
\item
{\em Independent V\&V}  (IV\&V)
is performed by an external team that, potentially,   explores very different
issues to the embedded V\&V team\footnote{Annex C of the IEEE 1012-2004~\cite{1012} 
standard
offers a more detailed definition of IV\&V. That definition retains Wallace \& Fujii's essential 
distinction between internal (embedded) and external (independent) V\&V.}.
\ei
Previously, we have offered guidance to IV\&V teams via issue predictors learned from
static 
code measures~\cite{me07c}.  Such predictors can be used
only {\em after} the code has been written.  Since it is much cheaper to
fix errors {\em before} code
generation~\cite{boehm88,dabney07,shu02,wallace89,arthur99}, 
this article
seeks early life cycle issue predictors.
%Much prior research has offered early life cycle predictors for later life cycle project
%issues. Some of that work
%was based on somewhat vague qualitative knowledge (e.g.~\cite{madachy97,me03i}) or a Delphi-style
%analysis where the authority of the conclusions relies on the authority of the authors (e.g.
%\cite{boehm01}.
%\cite[p254-268]{boehm00b}). 
%In some cases, researchers publish the data they use to make their predictors (e.g.~\cite{takagi05})
%but this is very rare. Even when   researchers might offer predictors for problematic projects, they
%are silent on how to
%handle those problems by, say, adjusting a project's
%IV\&V plan (e.g.~\cite{jiang02,ropp00}).
%

We show here that a predictor for issue frequency and severity can be built 
in a {\em reproducible} way
from public domain data\footnote{\protect\url{http://promisedata.org/repository/mb2}}
and 
software\footnote{The WEKA data mining workbench~\cite{witten05}}. 
The predictor is {\em operational}; \begin{figure}[!t]
\begin{center}
\includegraphics[width=1.9in]{silap.pdf}
\end{cener}
\caption{SILAP2.
}\label{fig:silap}
\end{figure}
i.e. it 
can be executed to generate precise statistics on its performance.
For example, we
show below that for NASA IV\&V data, the model is remarkably effective
at 
distinguishing between projects that will or won't produce
numerous high severity risks.  

Our proposed method extends SILAP ({\em Software Integrity Level Assessment Process}), 
an existing NASA IV\&V task selection method~\cite{costello05,me06b}.
As shown in \fig{silap},
SILAP2 requires some {\em local knowledge} of a project:
specifically, (a)~the total {\em budget}; (b)~the
\begin{figure}[!b]
{\scriptsize
\begin{tabular}{|r@{~=~}p{2.5cm}|r@{~=}p{4.1cm}|}\hline
\multicolumn{2}{|c|}{Derived}&
\multicolumn{2}{|c|}{Raw features}\\\hline
	co&Consequence&

am&Artifact Maturity\\

	dv&Development&


as&Asset Safety\\

	ep&Error Potential& 

cl&CMM Level\\
	pr&Process&
cx&Complexity\\
	sc&Software Characteristic& 
di&Degree of Innovation\\

\multicolumn{2}{|c|}{~}&do&Development Organization\\
\multicolumn{2}{|c|}{~}&dt&Use of Defect  Tracking System\\
\multicolumn{2}{|c|}{~}&	ex&Experience \\
\multicolumn{2}{|c|}{~}&	fr&Use of Formal Reviews\\
\multicolumn{2}{|c|}{~}&	hs&Human Safety\\
\multicolumn{2}{|c|}{~}&	pf&Performance\\
\multicolumn{2}{|c|}{~}&	ra&Re-use Approach\\
\multicolumn{2}{|c|}{~}&	rm&Use of Risk Management System\\
\multicolumn{2}{|c|}{~}&	ss&Size of  System\\
\multicolumn{2}{|c|}{~}&	uc&Use of Configuration Management\\
\multicolumn{2}{|c|}{~}&	us&Use of Standards\\\hline
\end{tabular} }
\caption{SILAP's 
{\em derived features} 
are computed from the {\em raw features}.
}\label{fig:silapFeatures}
\end{figure}
{\em costs } associated with applying various IV\&V{\em tasks} 
which
depends on domain details such as the skills
and fees of sub-contractor, etc;
and
(c)~the project {\em components},
i.e. large sections of the project  that are often
treated separately to the whole.
\input{tim/silapModel}
This local knowledge is then mapped into the {\em general knowledge}
described in this paper.
Each project 
{\em component} is described using the 
{\em features} described in \fig{silapFeatures}.
Each feature is scored  using the scales
described in the appendix. 
The features are then converted to
{\em error potential} and {\em consequence} scores using 
SILAP and the 
rules found by this paper. 
These scores are then passed to the task selection table
of \fig{silapModel} which proposes a set of {\em tasks} for
each {\em component}.
Each proposed task can be {\em cost}ed (via local knowledge)
and {\em rank}ed 
(via historical records of task cost effectiveness).
If the available {\em budget} is insufficient to complete all the {\em tasks},
then some can be {\em pruned} away.


\begin{figure*}
\scriptsize
\begin{center}
\begin{tabular}{r|r|r@{~}l|r@{~}l}
\multicolumn{2}{c}{~}&\multicolumn{4}{c}{$F_S$ = number of issues found of severity $S$}\\\cline{3-6}
phase& $S$ = severity& \multicolumn{2}{l}{9 robotic missions}& \multicolumn{2}{|l}{7 human-rated missions
}\\\hline
      2.  concept&                        1&           0&                  & 59& \rule{3mm}{2mm}\\
             &                        2&           7&   & 81&\rule{8mm}{2mm}\\
             &                        3&          54&   \rule{3mm}{2mm}&  168& \rule{9mm}{2mm}\\
             &                        4&          11&   & 0&\\
             &                        5&           2&                  & 0&\\\cline{3-3}\cline{5-5}
             &                     $\beta_2=\sum_{S=1}^5 F_S\cdot 10^{5-S}$    & 12,521           &                  &687,800  & \\\hline
 3. requirements&                        1&           1&                  &  291&\rule{15mm}{2mm}\\
             &                        2&          70&   \rule{4mm}{2mm}&  313&\rule{16mm}{2mm}\\
             &                        3&         874&  \rule{44mm}{2mm}&  430&\rule{21mm}{2mm}\\
             &                        4&         435&  \rule{21mm}{2mm}& 0&\\
             &                        5&         133&  \rule{7mm}{2mm}& 0&\\\cline{3-3}\cline{5-5}
             &                     $\beta_3=\sum_{S=1}^5 F_S\cdot 10^{5-S}$    & 171,883           &                  & 3,266,000 & \\\hline
    4.   design&                        1&           0&                  & 11&\\
             &                        2&          18&   \rule{1mm}{2mm}& 16&\rule{1mm}{2mm}\\
             &                        3&         275&  \rule{13mm}{2mm}& 96&\rule{5mm}{2mm}\\
             &                        4&         129&  \rule{7mm}{2mm}& 0&\\
             &                        5&          25&   \rule{2mm}{2mm}& 0&\\\cline{3-3}\cline{5-5}
             &                     $\beta_4=\sum_{S=1}^5 F_S\cdot 10^{5-S}$    & 46,815           &                  & 135,600 & \\\hline
  5.  implement&                        1&           1&                  & 51&\rule{3mm}{2mm}\\
             &                        2&         59 &   \rule{3mm}{2mm}& 78&\rule{4mm}{2mm}\\
             &                        3&         364&  \rule{18mm}{2mm}& 333&\rule{16mm}{2mm}\\
             &                        4&         282&  \rule{24mm}{2mm}& 0& \\
             &                        5&          76&   \rule{4mm}{2mm}& 0 &\\\cline{3-3}\cline{5-5}
             &                     $\beta_5=\sum_{S=1}^5 F_S \cdot 10^{5-S}$    & 108,296           &                  & 621,300 & \\\hline
   6.      test&                        1&           1&                  & 15 &\rule{1mm}{2mm}\\
             &                        2&          10&   & 27 &\rule{3mm}{2mm}\\
             &                        3&         361&  \rule{18mm}{2mm}& 77 &\rule{4mm}{2mm}\\
             &                        4&         271&  \rule{14mm}{2mm}& 0 &\\
             &                        5&          41&   \rule{2mm}{2mm}& 0 & \\\cline{3-3}\cline{5-5}
             &                     $\beta_6=\sum_{S=1}^5 F_S \cdot 10^{5-S}$    & 58,851           &                  &184,700  & \\\hline
\end{tabular}
\end{center}
\caption{A sample of issues found by NASA's IV\&V team.}\label{fig:issues}
\end{figure*}


Since the features of \fig{silapFeatures} can be collected early in a project's
life cycle, SILAP2 can be quickly applied to new projects.
To the best of our knowledge, this is the first report of
an issue frequency and severity
predictor that:
\bi
\item is based on public domain data;  and
\item
has a precise operational implementation; and
\item 
has a demonstrably useful performance; and
\item
can be tuned to local data; and
\item
most importantly,
can be applied early in the life cycle.
\ei

The rest of this paper is structured as follows. Before presenting methods for tuning IV\&V to particular projects,
we make two digressions. First, using a wide range of formal and informal evidence,
we make the case that IV\&V is worthy of study. 
Second, a literature review will explore what little is known of current IV\&V tasks.
This will be followed by a general discussion of data mining methods for SE data
and a specific discussion of what was learned by those data miners from NASA IV\&V data.
The paper ends with some  comments on external validity.

%\section{SILAP2}
%
%Perhaps it is time to change the focus of SE research from new task generation to em
%
%performance
%Unlike prior work, the conclusions of this method
%are (a)Such a predictor can be built using data from NASA's software Independent Verification
%\& Validation facility.
%Wallace \& Fujii~\cite{wallace89}  
%distinguish between:
%\bi
%\item {\em Embedded V\&V}, which  is done in-house by the developers or the users; 
%\item
%And {\em independent V\&V}  
%which is done externally by a group that might choose to explore very different
%issues to the embedded V\&V team\footnote{Annex C of the IEEE 1012-2004~\cite{1012} 
%standard
%offers a more detailed definition of IV\&V. That definition retains Wallace \& Fujii's essential 
%distinction between internal (embedded) and external (independent) V\&V.}.
%\ei
%This report audits and updates the current version of SILAP.  
%Budgen~\cite{budgen06} challenges software engineering researchers to better document the benefits
%of their preferred methods.
%In the Internet age, he
%argues, many sources of supposed knowledge
%(Google, Wikipedia, digg.com,
%and so on)
%surround us.  He poses
%the following challenge: 
%``how
%should we train students to assess all that information and to
%separate the sense from the nonsense?''. In his view, before we can
%denounce some inaccuracy in, say, Wikipedia, we must first look to
%our own work and audit our own results.  
%
%Accordingly, this article explores what we 
%{\em say}, rather that just we can {\em claim},
%about software independent verification \& validation (IV\&V). 
%

%Building the predictor was surprisingly difficult. 
%This is an open research question. The state of the research is
%typically a study of one task applied to one project. It is rare to find an empirical
%study
%comparing the
%cost-effectiveness  of multiple tasks over multiple projects. 


%
%NASA's current IV\&V program. Now, nearly 
%
%One proponent of evidence-based software engineering
%is David Budgen of Durham University. In the Internet age, he
%argues, many sources of supposed knowledge
%(Google, Wikipedia, digg.com,
%and so on)
%surround us.  He poses\footnote{
%Keynote address, CSEET'06. 
%} the following challenge: 
%``how
%should we train students to assess all that information and to
%separate the sense from the nonsense?''. In his view, before we can
%denounce some inaccuracy in, say, Wikipedia, we must first look to
%our own work and audit our own results.  
%
%The time is right for this article.
%Based on NASA's experience,
%two issues will be explored:
%\bi
%\item
%{\em Identifying projects that need more IV\&V:}
%What are the minimum set of project characteristics that most predicts for
%a problem software project?
%\item
%{\em Conducting IV\&V in a cost-effective manner:}
%What are the minimum set of cost-effective IV\&V tasks that most finds specific issues
%in software?
%\ei
%We show that it is possible to build
%effective predictors for projects that
%need more IV\&V.
%{\em Screening experiments} will be conducted,
%that reduce a larger set of factors
%to just those with
%an
%important effect on the severity levels of issues found by IV\&V.
%The simplicity of the predictors will be surprising. Several factors
%that the literature suggests {\em should} predict for
%greater defects are absent from our learned predictors.
%
%We also show that
%the list of most cost-effective IV\&V tasks
%can be readily found.
%This list will be quite small.
%suggesting that many of the  current lists of proposed
%IV\&V tasks (e.g.~\cite{1012,costello05,wallace89}) are
%overly elaborated.
%
%This specific conclusions of this study are two-fold.
%\be
%\item
%NASA has 
%has been consistently {\em under-estimating}
%the risks associated with NASA software projects. 
%\item
%IV\&V is often applied to static code artifacts.  
%We show here the
%analysis of such late life cycle artifacts is one of the least
%cost-effective IV\&V methods.  
%The {\em effectiveness} of early life cycle  V\&V
%has been made before (e.g.\cite{boehm88,dabney06a,shu02,wallace89,arthur99}).
%However, to the best of our knowledge,  this is the first clear result relating to
%the relative {\em ineffectiveness} of later life cycle V\&V.
%\ee
%
%\noindent
%There are two business-level implications of the above:
%
%\bi
%\item
%NASA must expand its IV\&V program.
%\item
%All organizations, not just NASA, should
%{\em reduce} late life cycle verification-based
%IV\&V (that focuses on code artifacts) and {\em increase}
%early life cycle validation-based IV\&V (that focuses
%on the business context of a project).
%\ei
%

%Prior experience with projects resulted in the experience factory approach developed at NASA SEL.
%A premise of that approach was enough organization stability to support a unified, centralized
%reference model for software development.
%That stability does not currently exist at NASA and we cannot expect such stability to appear
%in the foreseeable future. 
%
%However, one of the NASA SEL lessons still applies: 
%data collection  must be compromise in asking for only as much information as is feasible to obtain.
%This study lacked the resources to define a new data collection method. Rather, it relied on the
%data currently available within the databases of the NASA IV\&V
%facility. 
%
%Due to the pace of organizations change with NASA, and within its databases,
%very little of the 15 year track record of NASA IV\&V was available to this project. The best 
%we could do was to assemble 211 records mapping the SILAP1 raw features (collected early in the IV\&V process)
%to the severities of the issues found by IV\&V. This lack of data may seem surprising, but it is quite
%typical of highly dynamic organizations such as NASA.
%
%
%Based on this experience, we offer the following conclusions.
%
%
%The contribution of this paper is two-fold. Firstly, we show that
%in a few days it is possible to characterize the software elements
%of a project in sufficient detail to build a predictor for the kinds
%of issues this project
%
%features of a software
%project
%The rest of this   paper is structured as follows:
%\bi
%\item
% After documenting
%the benefits of IV\&V, a literature review will show
%that few papers empirically compare a large range of IV\&V tasks.
%\item
%In order to collect new data, two alternate approaches will be discussed:
%heuristic data mining and a NASA SEL-style ``learning organization'' method.
%Given the current pace of change within NASA, this study resorts
%to heuristic data mining.
%\item
%Since it can't collect {\em new} data,
%this  study will have to use the {\em available data}, none of which was
%collected especially for this purposes. Hence, like many other
%real world SE data sources, it contains much noise (signals not connected
%to the target). Various noise-reduction methods will be discussed including
%RIPPER~\cite{cohen95r} and the WRAPPER~\cite{kohavi97}. 
%\item
%These methods are then
%applied to NASA's current IV\&V task selection method, SILAP. Drawbacks
%with SILAP will motivate a new method, SILAP2. 
%\item
%Empirically, SILAP2 performs very well.
%In a 10-way cross validation experiment, SILAP2
%is nearly perfect in predicting for issues severity (for four classes of severity,
%the $f$ measure was 0.95,0.96,0.95, and 0.94).
%\item
%In a small number of cases, SILAP2 recommends more IV\&V tasks than what is strictly necessary.
%Hence, SILAP2 is augmented with {\em pruning} heuristics 
%describing what tasks to ignore, if the available budget cannot accommodate
%all the selected tasks. 
%\ei
%


\section{Benefits of IV\&V}

\PARstart{T}{here} is much formal and informal evidence for the value of IV\&V (by ``informal'' evidence, we mean that evidence 
that is
{\em not} published in sources
that  other researchers can verify).
Interviews with NASA IV\&V personnel
has yielded the  following informal evidence
for the benefits of IV\&V:
\bi
\item
``Even when developers assess their own products, they focus
on a very small part of a much larger system. IV\&V workers, on the other hand,
can explore a larger area.''
\item
``IV\&V has become NASA's corporate memory for software-related
issues.
Development teams can change personnel faster than the IV\&V team
so  IV\&V can come to understand the software as well or better
than the development team. 
For example, in several NASA multi-year projects, most of the issues
were found
by the IV\&V team, and not the developers.
 Also, in numerous cases, the IV\&V team can recall  why (e.g.)
three years ago, some crucial
decision
was made.''
\item
``The technical skill of NASA's IV\&V's teams are acknowledged by
(some) of NASA's contractors.
In one case, the prime contractor 
voluntarily gave up funding
within its own budget so that the NASA project
office could pay for more IV\&V,
over and above the amount originally planned for IV\&V.''
\ei

More formal evidence for the benefit of IV\&V comes from the 
literature and NASA technical reports.
Zelkowitz and Rus discuss the Space Shuttle IV\&V  program in the 1990s~\cite{zelkowitz01}. 
They report a large decrease in major flight defects 
once IV\&V was introduced, and   that IV\&V uncovered and resolved
certain
shortcomings in the shuttle's testing program. 
Almost half the issues found by
IV\&V were found very early in the life cycle of shuttle software upgrades.

\fig{issues} shows the issues found in recent
NASA missions by the IV\&V team. Note that (a)~most of the issues are found early in the life cycle
(in the concept and requirements phase) and (b)~many
of the issues
were of high criticality (severity 1 and 2), especially for the human-rated missions.

Economically, it is useful to find errors early.
Boehm \& Papaccio 
advise that reworking software is far
cheaper earlier in the life cycle than later ``by factors
of 50 to 200''~\cite{boehm88}.

\begin{figure}
\begin{center}
{\footnotesize
\begin{tabular}{r@{~}|r@{~}l@{~}|r@{~}l}
phase & \multicolumn{2}{c|}{IV\&V}&\multicolumn{2}{c}{no IV\&V}\\\hline
requirements& 16&\rule{8mm}{1.5mm}& 0&\\
high-level design& 20&\rule{10mm}{1.5mm}& 2 & \rule{1mm}{1.5mm}\\
low-level design& 31&\rule{15mm}{1.5mm}& 8&\rule{4mm}{1.5mm}\\
coding \& user testing& 24&\rule{12mm}{1.5mm}& 34&\rule{17mm}{1.5mm}\\
integration \& testing& 6&\rule{3mm}{1.5mm} & 14 & \rule{7mm}{1.5mm}\\\hline
totals& 97&& 58 & 
\end{tabular} }
\end{center}
\caption{Defects found
with and without IV\&V. From \protect\cite{arthur99}.}\label{fig:arthur}
\end{figure}

\begin{figure}
\begin{center}
{\scriptsize 
\begin{tabular}{r@{~}r|r@{~}r@{~}r@{~}r@{~}r@{~}r}
\multicolumn{2}{c}{~}&\multicolumn{6}{c}{Phase issue found}\\\cline{3-8}
&Phase issue        & f=1        & f=2   & f=3  & f=4 &f=5 &f=6\\
i&introduced   &Requirements& Design& Code &Test &Int& Operations\\\hline 
1&Requirements &           1& 5     & 10   & 50  & 130& 368\\
2&Design        &            &  1   & 2    & 10  &  26&  74 \\
3&Code          &            &      & 1    & 5   & 13 &  37\\ 
4&Test           &           &      &      & 1   & 3 & 7\\ 
5&Integration    &          &       &      &     & 1 & 3\\\hline
   &$\Delta$ = mean $\left(\frac{C[f,i]}{C[f,i-1]}\right)$
                   &&5          & 2    & 5   &2.7     &2.8\\            
\end{tabular} }
\end{center}
\caption{$C[f,i]$: The cost-to-fix escalation factors, relative to
fixing an issue in the phase where it was found ($f$) versus
in the phase where it was introduced ($i$).
Last row shows the cost-to-fix delta if the issue  introduced in phase $i$ 
is fixed immediately afterwards in phase $f=i+1$. From~\cite{dabney07}.}
\label{fig:dabney06a}
\end{figure}

This effect has been widely documented in other research.
A panel at IEEE Metrics 2002~\cite{shu02}.
concluded that
finding and fixing  severe software problems
after delivery is often 100 times more 
expensive than finding and fixing it during 
the requirements and design phase~\cite{shu02}. 
Arthur et.al.~\cite{arthur99} conducted a small controlled experiment
where
a dozen engineers at NASA's Langley Research Center were 
split into development and IV\&V teams. The same application
was written with and without IV\&V. 
\fig{arthur} shows the results:
(a)~more issues were found using IV\&V than without; (b)~the issues
were found much
earlier.
This is a compelling argument for IV\&V. 
Even if the IV\&V team found the {\em same} bugs as the development
team, but found them {\em earlier}, the cost-to-fix 
would be reduced by a significant factor.


Similarly, Wallace \& Fujii~\cite{wallace89} discuss 
studies on IV\&V started at different phases.
Starting  at the coding phase
saves 20\% of the V\&V costs.
However,
starting earlier at the requirements
phase saves 92\% to 180\% of the V\&V costs.

Finally, \fig{dabney06a} shows 
the cost of quickly fixing an issue relative to
leaving it for a later phase (data from four NASA projects~\cite{dabney07}). 
The last line of that table shows that
delaying issue resolution 
even by one  phase
increases the cost-to-fix of $\Delta=2 \ldots 5$.
Using this data, 
Dabney et.al.~\cite{dabney07} calculates 
a dollar
spent on IV\&V 
returns to NASA \$1.21, \$1.59, \$5.53, and \$10.10 (on four NASA projects). 
\begin{figure*}
\scriptsize
\begin{center}
\begin{tabular}{r|r@{~}l|p{2.75in}|l}
year & & \#IV\&V project at Fairmont IV\&V &   notes & oversight\\\hline
1988 &n/a & &Space shuttle begins IV\&V&Johnson Space Flight Center (Texas)\\\hline
1991 &n/a& &Congress pass bill creating the IV\&V facility in Fairmont, West Virginia & NASA headquarters (east  coast)\\
1994 &1  &\rule{1mm}{1.5mm} &International space station IV\&V at Fairmont begins &\\
1995& 1 &\rule{1mm}{1.5mm}&&\\\hline
1996& 2 &\rule{2mm}{1.5mm}&&NASA Ames (west coast)\\
1996& 3 &\rule{3mm}{1.5mm}&New contracts enables IV\&V for all NASA projects
        (but projects must fund IV\&V from their own line items) &\\
1997& 3 &\rule{3mm}{1.5mm}&&\\
1998& 3 &\rule{3mm}{1.5mm}&&\\
1999& 12 &\rule{12mm}{1.5mm}& NASA mandates that IV\&V be considered for all NASA software&\\\hline
2000& 15&\rule{15mm}{1.5mm}& Facility oversight moves to Goddard Space Flight Center& GSFC (east coast)\\
2001& 20 &\rule{20mm}{1.5mm}&&\\
2002& 36&\rule{36mm}{1.5mm}&&\\
2003& 42 &\rule{42mm}{1.5mm}&IV\&V now funded from a central agency source. &\\
2004& 37&\rule{27mm}{1.5mm}&&\\
2005& 24&\rule{24mm}{1.5mm}&SILAP data collection begins&\\
2006& 26&\rule{26mm}{1.5mm}&&\\
2007& 24&\rule{24mm}{1.5mm}&&\\
\end{tabular}
\end{center}
\caption{History of NASA IV\&V.}\label{fig:hist}
\end{figure*}



\section{Related Work}
\PARstart{H}{aving} documented the benefits
of IV\&V, the next question a manager might ask is ``how to do it?''.
This is a difficult question to answer.
The literature describes nearly a hundred V\&V and IV\&V tasks:
\bi
\item
Wallace \& Fujii offer 20 essential and 36 optional
tasks for software V\&V~\cite{wallace89}. 
\item
Table 2 of the  IEEE-1012 standard for software V\&V
offers a list of 52 minimal tasks for IV\&V. Each task is selected by a combination of development phase and
estimated severity in project errors~\cite{1012}.  
\item
\fig{silapModel} offers  40 pre-deployment
IV\&V tasks.
\ei

In the literature, there are very few attempts to empirically assess the relative
cost-effectiveness of this broad range of IV\&V techniques.
Most of the reports
take the form of (e.g.)
Easterbrook et.al.~\cite{easter98} who discuss the merits of using a single
lightweight formal method for IV\&V 
requirements models on one NASA project.
Another study worthy of mention is
that of  Hayes, Dekhtyar, and Sundraram who explore
the relative merits of
different tools for finding links between requirements documents~\cite{hayes06}. That analysis includes a comparison
of human expert and state-of-the-art commercial tools and a new research prototype.
For more examples of singleton evaluations of one method on one project,
see
the proceedings of the annual
symposium 
Program\footnote{\url{http://sas.ivv.nasa.gov/conclusions/index.php}} and
the  repository of NASA's Software Assurance Research\footnote{\url{http://sarpresults.ivv.nasa.gov}}.

Several researchers offer detailed taxonomies of standard NASA 
faults~\cite{hayes05,malin07},
but
do not address what tasks are most cost-effective at finding the faults. 
For example,
Lutz reports
studies on the most frequent kind of faults seen in deep space NASA missions but does
not fully address the issue of what V\&V and IV\&V tasks are best at removing faults~\cite{lutz04}. 

Leveson talks in general terms
about how to increase the safety of software~\cite{leven95}.
Her preferred methods would significantly change
current IV\&V practice~\cite{levenson96} so it is difficult to map that advice into
selecting the IV\&V tasks of \fig{silapModel}.




In a limited number of publications, some attempt is made to generalize over
multiple projects, multiple methods, or projects extending over many years.
Those publications include:
\bi
\item
Madachy's  heuristic software risk model that alerts when certain 
observations are seen in a project~\cite{madachy97};
\item
Boehm and Basili's ``top 10''  defect reduction list~\cite{boehm01,shu02};
\item The Dabney study, based on  four NASA projects~\cite{dabney07};
\item Menzies et.al.'s discussion of 
learning predictors for software defects
using data from five NASA projects~\cite{me03q,me07b}; or
methods for generating cost models from 25 software projects, including 11
 NASA developments~\cite{me06d};
\item
Glass' list of six factors seen in 16 ``runaway'' software projects;
i.e. projects where
``schedule, cost, or functionality that was twice as bad as the original estimates''~\cite{glass97}; 

\item Raffo et.al.'s software process of a standard waterfall model, modified to handle IV\&V tasks~\cite{raffo04}.
\item
Jiang et.al.'s
table of the connection of
40 project features to the software risks seen in 83 projects~\cite{jiang02}.
\item
Ropponen \& Lyytinen
analysis of questionnaires from 83 project managers and 1,110 projects that identifies
26 software risk components~\cite[p110]{ropp00};

\item
Takagi \& Abe et.al.'s study of 31 projects 
that identifies features that predict
for software ``runaways''~\cite{takagi05,abe06}
\ei
For our purposes, these studies are incomplete.
Some of this research bases its conclusions on  just a few projects~\cite{glass97,dabney07,me03q,me07b};
Others use a Delphi-style analysis
 were the authority of the conclusions
rests on the authority of the author~\cite{boehm01,shu02,madachy97}
Alternatively, they may
base their conclusions on extensive data collection
but, for confidentiality reasons, can not
make that data available for public scrutiny~\cite{jiang02,ropp00,raffo04}.
It is hard to make critical audit conclusions  based on a Delphi-style analysis or
inaccessible data.
In general, only a minority of SE researchers can
publish the data used to make their conclusions (for example,~\cite{me06d,takagi05,abe06,me03q,me07b}).


Also, several of the these papers~\cite{jiang02,ropp00} only
offer general advice about how to avoid problems in software development.
The subjective nature of this advice makes it difficult to consistently
deploy
them 
over a national software development program.
Only a few~\cite{me06d,me07b,abe06,takagi05} 
offer public domain versions of their data and models.

Note that,  even in combination, the above publications cover a small
fraction of the (I)V\&V tasks listed in the literature.


\section{Data Mining}

\PARstart{S}{ince} existing sources were not informative,
this study
applied
data mining to 40 IV\&V tasks used in
211 NASA components from 6 projects.

\subsection{Issues with Data Collection}

It may surprise the reader that
after 19 years, this study could only access 211 components.
After all, as shown in \fig{hist}, NASA has a 19 year history of IV\&V.

What must be understood is that NASA is similar to many
other large organizations that constantly adapt their practices
and their databases as their
business changes. 
Various factors have resulted in a steady pace of change at NASA:


%Data mining can be characterized as the heuristic exploration data of the available data.
%That data is expected to be noisy (i.e. contain signals not related to the target concept)
%so various methods are applied to scrub the data and prune superfluous parts of the learned
%model. 
%
%An alternate strategy, that avoids noisy data, is to design a data collection
%program that is certain to collect consistent data relating to a specific business goal.
%This alternate strategy was used
%at the NASA software engineering laboratory (SEL).
%Researchers at the SEL
%argued for the careful
%design of metrics in order to answer questions relating to significant business 
%goals~\cite{basili02,basili88,basili99}.
%In this ``learning organization'' 
%a {\em quality improvement paradigm} is applied the 
%scientific method to software engineering to
%create an
%{\em experience factory} 
%that systematically saved and reused experience from previous projects.  
%
%The SEL's learning organization experiment lasted ten years (from 1984 to 1994)
%during a
%period of relative stability within the NASA organization. 
%Starting in 1995, the pace of change within NASA increased.
%New projects were often out-sourced and the SEL became less the
%driver and more the observer, less pro-active and more reactive
%to software development activities. Without an owner of
%the software development process, each project could adopt its
%own structure. SEL-style experimentation became difficult since
%there was no longer a central model to build on.
%Basili et.al.~\cite{basili02} describes this 1995-2001 phase as one
%of ``retrenchment'' at the SEL.
%
\bi
\item
In the 1990s, NASA moved to  a ``faster, better, cheaper'' paradigm that
led to massive changes in NASA's development practices. For example, prior to 1990, NASA's Jet Propulsion
Laboratory launched (on average) five spacecraft a year, all of which were built internally. In the 1990s,
JPL launched dozens of craft, most of which were built by external contractors. 
This policy led
to certain high profile errors which were attributed to production haste, poor communications, and
mistakes in engineering management~\cite{davis00}. Accordingly, 
NASA terminated  ``faster, better cheaper'' 
and changed development methods. 
\item
In 2003, the loss of the Columbia orbiter during re-entry prompted
another major agency reorganization. Development budgets were significantly altered to fund the ``return to flight''
program that led to the successful re-launch, in 2005, of the shuttle program.
\item
Currently,  NASA is significantly reorganizing  (again) around the new  ``vision for space exploration''.
\ei
NASA's numerous national reorganizations  have been reflected in the
IV\&V program.
\fig{hist} shows that, four times,
the oversight for IV\&V has passed
between various NASA centers and NASA headquarters.
Also,
in 1996, 
NASA developed new contractual methods that allowed IV\&V to be
applied to a wide range of projects.
These new methods resulted in  a large change to IV\&V management practices
as the focus expanded from a few on-going projects to dozens of smaller
projects.

In 1999, after over a decade
of success,
IV\&V was mandated
to be considered for all NASA software.
But a problem remained-
projects had to find funds for IV\&V from
within their own budgets. This  often slowed IV\&V
work since
managers were forced to juggle the demands of
developments with the demands of IV\&V.
It took four more years of successful IV\&V before NASA
could solve this problem.  In 2003, NASA reaffirmed and strengthened
its support for IV\&V by
establishing IV\&V as an Agency-level program funded by the Agency
rather than by each individual project separately.

The above changes meant that more projects could receive the benefits of IV\&V.
However, the same changes that benefited NASA also
complicated
the twin goals of
data collection and analysis.
In nearly all projects,
the schema of IV\&V's issue tracking
systems was 
modified to better address some specific requirement.
While some data fields were required,
most 
were not suitable for comparing
the cost-effectiveness of IV\&V across projects.
Where possible, database schemas are standardized across multiple projects and multiple years
(e.g. SILAP data has been collected since 2005).
However,  
no data is collected consistently across all projects and all years.

Consequently,
this study had to make do with the available data, none of which was collected especially
for this purpose. Hence, 
it contains much ``noise'' (spurious signals not necessarily connected to the target
of predicting issue frequency and severity).

\subsection{Handling Noisy Data}

When learning from noisy sources, it is important to use methods that aggressively
prune noisy signals such as: 
\bi
\item
The RIPPER~\cite{cohen95r} rule learner;
\item
The WRAPPER algorithm~\cite{kohavi97} 
that explores subsets of the
available features. 
\ei

Both methods are standard techniques for removing noise and
generating succinct, easily explainable,
models from data.
RIPPER
takes a particularly aggressive approach to pruning away superfluous parts of a model:
\bi
\item 
After building a {\em rule}, RIPPER performs a back-select to see what rule
conditions
can be deleted, without degrading the performance of the rule.
\item
Similarly,
after building a {\em set of rules}, RIPPER performs a back-select to see what {\em rules}
can be deleted, without degrading the performance of the rule set.
The learned rules are built while minimizing  their {\em description length}.
This is an information theoretic measure computed from the size of the learned rules,
as well as its errors. If a rule set is over-fitted, the error rate increases,
the description length grows, and RIPPER applies  rule set pruning.
\item
RIPPER also employs a novel
{\em rule set optimization} algorithm that tries
replacing rules
with some straw-man alternatives (i.e. rules grown very quickly).
\ei

RIPPER is one of the fastest rule learners currently known in the literature. In its
haste, it may miss parts of the model that can be pruned. Hence, a much slower
and more thorough feature pruner was also employed.
Kohavi \& Johns' ``WRAPPER'' algorithm~\cite{kohavi97}
explores subsets of the available features. Each subset is assessed by
asking some target
learner (in our case, RIPPER)
to build a model using just that subset.
The WRAPPER grows the feature
set until the target learner reports that larger sets do no better 
than subsets.
The algorithm stops when there are no variables left,
or there has been no significant improvement in 
after the last five additions (in which case, those last five
additions are deleted). Technically, this is a hill-climbing forward select
with a ``stale'' value of 5.
Empirically, WRAPPER is known to comparatively slower than
other 
other feature pruners, 
but yields best results~\cite{hall03}.

\subsection{Modeling the Dependent Feature}

RIPPER inputs ``feature vectors''; i.e. sets of independent features plus one
dependent feature called the {\em class}. The goal of the learning is to find
some combination of the independent features that predict for the dependent class feature.

In this study, the dependent feature was the number of issues found 
in a component,
weighted by the severity.
NASA now uses a standard five-point scale  to score issue severity.  A
slightly different scale is used for robotic and human-rated missions
(see \fig{robots} and \fig{humans}) but the interpretation of the
scales is same across all missions: severity one issues imply total
disaster while severity five issues are ignorable.

\input{tim/severity}
For the purposes of analysis,
the severity counts were summarized 
as follows. Let $s_{i,j}$ be the frequency of severity issue $i$ in component
$j$. If $Max_i$ is the maximum number of issues of severity $i$, totaled
across all components
then $W_j$ is a weighted sum of all the severities.

The modeling intent of $W1_j$
was that
any higher-level severity report  is  more important than any number
of lower level severities. In this interpretation, 
issue reports are {\em thresholds} that must be reached
in order to {\em trigger} different activities.
To implement this, 
the severity counts $s_{i,j}$ is divided by the maximum count for severity $i$:
\begin{equation}\label{eq:w1}
W1_j= 1 + \sum_{i=1}^5\left( \frac{s_{i,j}}{1+Max_i}\right){\cdot}10^{6-i}
\end{equation}

(The 
``plus one'' parts of these equations were added to avoid numeric errors. 
In this data, the severity frequency counts were
large enough to make the ``plus one'' contributions negligible.)



\subsection{Modeling the Independent Features}
A learned model predicts for the dependent feature using some combination
of independent features. The independent features used
in this study came from NASA's SILAP model~\cite{costello05,me06b}.

Since 2005, 
NASA IV\&V has been collecting
a standard set of project features
at the start of every IV\&V project. 
SILAP uses these features to predicts for error potential (a.k.a. likelihood) and 
consequence of error (a.k.a. criticality).

SILAP was built via Delphi-sessions amongst NASA civil servants experienced with IV\&V.
This model holds
the group's consensus belief on what affects the quality of NASA projects.
\fig{silapFeatures} showed the
SILAP ``derived'' factors that are computed from the ``raw'' factors using the functions of \fig{SILAP.awk}.
These computed error potentials and consequences were then used to select IV\&V tasks, using
\fig{silapModel}.

 SILAP grew into an early life cycle project assessment methodology for IV\&V.
NASA civil servants 
report that SILAP has greatly clarified their discussions
with projects regarding what IV\&V tasks are/are not to be performed.
Since SILAP was built locally at
IV\&V, it is
understood locally.
Previous risk models, which had been developed elsewhere,
were to a degree ``black box'' models that were 
poorly understood. Hence, it was harder to defend their conclusions
to clients lobbying
for less IV\&V oversight.

Each raw SILAP factor is weighted and, usually, filtered by a second numeric to derive
the consequence and error potential values.
The {\em weight*filter} column of \fig{cons} shows the relative contribution of each raw
factor, as found by the Delphi sessions amongst the NASA civil servants.
\begin{figure}[!t]
{\scriptsize
\begin{alltt}
function CO( tmp) { 
        tmp=0.35*AS + 0.65 *PF; 
        return (round((HS) > tmp) ? HS : tmp) } 
function EP() {table
        return round(0.579*DV() + 0.249*PR() + 0.172*SC()) } 
function SC() { 
        return 0.547*CX + 0.351*DI + 0.102*SS } 
function DV() { 
        return 0.828*EX + 0.172*DO } 
function PR() { 
        return 0.226*RA + 0.242*AM + formality() } 
function formality() { 
        return 0.0955*US + 0.0962*UC + 0.0764*CL  +
               0.1119*FR + 0.0873*DT + 0.0647*RM }
\end{alltt}}
\caption{SILAP.awk: computes the derived features from the raw features of \protect\fig{silapFeatures}.}\label{fig:SILAP.awk}
\end{figure}
\begin{figure}
\begin{center}
\footnotesize
\begin{tabular}{p{1.2cm}|rrrr|r}
& & & & weight*& contribution\\
goal & feature & weight & filter & filter & to goal\\\hline
    consequence&      hs&   0.35&      1&0.350& 35\%\\
     (co)&      pf&   0.65&      1&0.650& 65\%\\\hline
     error&      ex&  0.828&  0.579&0.479& 47\%\\
     potential&      cx&  0.547&  0.172&0.094&  9\%\\
     (ep)&      do&  0.172&  0.579&0.100&  9\%\\
     &      di&  0.351&  0.172&0.060&  6\%\\
     &      am&  0.242&  0.249&0.060&  6\%\\
     &      ra&  0.226&  0.249&0.056&  5\%\\
     &      us& 0.0955&  0.249&0.024&  2\%\\
     &      uc& 0.0962&  0.249&0.024&  2\%\\
     &      fr&  0.119&  0.249&0.030&  2\%\\
     &      dt& 0.0873&  0.249&0.022&  2\%\\
     &      ss&  0.102&  0.172&0.018&  1\%\\
     &      cl& 0.0764&  0.249&0.019&  1\%\\
     &      rm& 0.0647&  0.249&0.016&  1\%
\end{tabular}
\end{center}
\caption{Relative contribution of \protect\fig{SILAP.awk} factors to the goals of consequence and error potential.
The percentages shown on the right are normalized values for the $weight*filter$ column. }\label{fig:cons}
\end{figure}
%\begin{figure}
%\begin{center}
%\footnotesize
%\begin{tabular}{r|rr}
%\multicolumn{1}{c}{~}&\multicolumn{2}{c}{correlation, feature to $W_x$}\\\cline{2-3}
%feature&$W_1$ (from \eq{w1}) & $W_2$ (from \eq{w2})\\\hline
%consequence (co) & -0.19 &-0.15\\
%error potential (ep) & 0.59& 0.61\\
%co * ep & 0.07 & 0.15
%\end{tabular}
%%\begin{center}
%%\includegraphics[height=2.25in]{tim/coepissues3D.pdf}
%%\includegraphics[height=1.5in]{tim/coepissuesCOW.pdf}
%%\includegraphics[height=1.5in]{tim/coepissuesEPW.pdf}
%\end{center}
%\caption{Correlation between weighted scores ($W_x$) and derived
%SILAP features error potentials (ep), consequence (co)
%seen in 211
%NASA sub-systems from 6 systems. 
%}\label{fig:epcow}
%\end{figure}
%

This study could access SILAP descriptions of 280 components  from
11 NASA projects. All these projects received IV\&V in the period
2005 to 2007.
Of these, only six of those projects had an issue tracking data
schema
that could generate a consistent set of issue data.
Therefore this study was based on data from 211 components  taken
from 6 projects.  None of these projects were human-rated missions.
It was hoped that more data would be available but, given the pace
of change within NASA databases, this was not the case.

Each component  was described  using:
\bi
\item
the 16 raw SILAP features of \fig{silapFeatures}; 
\item
five issue severity counts $s_1,\ldots s_5$; 
\item
the project name: $L1$; and
\item
 the component  type: $L2$; i.e.  GNC (guidance, navigation,
and control);
PAYLOAD (specialized
payloads); CDH  (command \& data handling), and GROUND (ground systems). 
\ei
Note that the relationship of projects to components is one to many; i.e.
the 211 components exist in one, and only one, of the six projects.
Also, the relationship of components  to component types is many to one;
i.e.
the 211 components 
are each assigned one, and only one, of the four component types.


%.
%.\fig{epcow} shows the relationship between the outputs derived
%.by SILAP1 (from \fig{SILAP.awk}) and the weighted sums $W1$ and $W2$ of the severity reports.
%.If SILAP1 was an accurate predictor for problems, we would expect some correlation between
%.$W$ and {\em either} 
%.consequence of error (co) {\em or} error potential (ep) {\em or}
%.their product (co*ep)\footnote{The correlation between two
%.array $X$ and $Y$ with means \bar{x} and \bar{y} (respectively) and
%.items $x\in X$ and $y \in Y$  is 
%.$\frac{\sum(x - \bar{x})(y- \bar(y)}{\sqrt{\sum(x - \bar{x})^2\sum(y - \bar{y})^2}}$.
%.}. Correlation ranges $-1 \le r \le 1$  and correlation values close to zero imply no connection
%.between the distributions being studied. Of the correlations  shown in \f
%.
%.%\begin{figure}
%.%\definecolor{gray}{gray}{0.8} 
%.%\definecolor{g}{gray}{0.81} 
%.%\scriptsize
%.%\begin{center}
%.%\begin{tabular}{r@{~}|r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r@{~}r}
%.%&as&pf&ex&do&us&uc&cl&fr&dt&rm&ra&am&cx&di&ss&$W$\\\hline
%.%hs&24&18&4&12&-10&34&-20&43&44&3&7&13&8&46&21&-21\\
%.%as &&48&-20&-7&-25&-4&-30&16&24&-10&0&-7&42&23&9&-25\\
%.%pf & &&-9&-16&-21&-5&-26&5&19&1&13&-2&44&11&12&-11\\
%.%ex & & &&-12&\colorbox{gray}{73}&29&\colorbox{gray}{67}&-27&-3&-28&-3&30&-29&33&29&\colorbox{gray}{68}\\
%.%do & & & &&6&9&4&30&-20&37&-23&4&-27&2&-23&-3\\
%.%us & & & & &&37&\colorbox{gray}{58}&-25&-4&-21&-22&11&-43&24&7&\colorbox{gray}{56}\\
%.%uc & &  & & & &&37&\colorbox{gray}{51}&41&13&-3&47&3&30&7&-18\\
%.%cl & & & & & & &&-26&-28&-7&-6&\colorbox{gray}{53}&-25&10&23&\colorbox{gray}{65}\\
%.%fr & & & & & & & &&35&32&31&0&20&12&-21&\colorbox{gray}{-66}\\
%.%dt & & & & & & & & &&2&-8&-14&35&31&23&-37\\
%.%rm & & & & & & & & & &&28&23&2&-20&-30&-18\\
%.%ra & & & & & & & & & & &&14&28&-7&11&-1\\
%.%am & & & & & & & & & & & &&4&12&14&17\\
%.%cx &  & & & & & & & & & & & &&4&27&-33\\
%.%di & & & & & & & & & & & & & &&31&13\\
%.%ss & & & & & & & & & & & & & & &&33\\
%.%\end{tabular}
%.%\end{center}
%.%\caption{Percent 
%.%correlations within 211 sub-systems
%.%between SILAP1 parameters scored 1,2,3,4,5 and the $W$ score from \protect\eq{w}.
%.%The nine pairs with \mbox{$correlation \le -50\%$ or  $correlation \ge 50\%$} are highlighted
%.%in a \colorbox{gray}{gray box}.
%.%}\label{fig:corr}
%.%\end{figure}
%.%
%.\fig{corr} raises other doubts about the efficacy of SILAP1.
%.Correlation\% varies from -100\% to 100\% and a correlation of zero means that there is no numeric
%.correspondence between two variables.
%.As shown in \fig{corr},
%.most of the correlations in our data have correlations close to zero.
%.In terms of building succinct models, this is a useful result since it suggests that
%.a predictor for issue severity might be built using just a small subset of the SILAP1 features.
%.We will return to this point, below.
%.
%.In only nine cases (those shown with gray boxes in \fig{corr}) 
%.are the correlations above 50\% or below -50\%.
%.Some of those correlations corresponds to established wisdom in the field. For example,
%.Basili \& Shull \& Lanubile~\cite{basili99}
%.would accept the strong correlation between increasing
%.the use of formal reviews (fr) and decreasing high severity issues (correlation=-66\%).
%.However, we know of no researcher who would accept the following adherent results:
%.\bi
%.\item
%.Increasing programmer experience (ex) 
%.{\em increases} high severity scores ($W$), correlation=68\%.
%.\item
%.Increasing CMM-level (cl) {\em increases} high severity scores ($W$), correlation=65\%.
%.\ei
%.
%.A review of our data  explains these adherent results. Recall that
%.\eq{w} weighted severity one errors by a factor of 100,000. Hence,
%.even a small system with just a few sub-systems
%.can have a disproportionate impact on the correlations
%.if that system has severity 1 issues.
%.This is indeed the case in our data.
%.

\subsection{Data Quirks}\label{sec:quirks}

Before applying automatic data mining, it is good
practice to inspect the data distributions
for any obvious anomalies.
In the independent data, for example, all the data ranges over
$1 \le x \le 5$
and there were no instances
with $uc\in \{4,5\}$ or $us \in \{4\}$ or $am \in \{4,5\}$.
Also, there were only $\frac{4}{211}$
examples of $ra \in \{2\}$. We can use this knowledge to
rewrite rule conditions like
$uc \le 1 \wedge ra < 3 \wedge am \ge 3$ into
a more {\em simplified form}  such as
$uc=1 \wedge ra=1 \wedge am=3$.

Also, in the dependent data, there are very few reports of severity
one issues.  In that project, the managers of that system were
highly concerned about quality. Hence, they employed numerous best
practices including high CMM practices ($cl$), extensive use of
standards ($us$), hiring very experienced developers ($ex$). That is,
these practices did not {\em cause} the severity one issues in that
project. Rather they were adopted {\em in response} to 
perceived issues with the project.

 
%.Based on the above, we  comment that:
%.
%.\bi
%.\item
%.\fig{epcow} shows that
%.SILAP1's calculated consequence and error potential values are poor predictors for
%.issue severity.
%.Hence, SILAP1 needs revision. 
%.\item
%.\fig{corr}
%.shows that only a subset of the SILAP1's features are connected to other features.
%.Hence, 
%.a predictor for issue severity might use only a subset of the
%.SILAP1's features. 
%.\item
%\ei

\section{Learning SILAP2}

\subsection{Pre-Processing}

RIPPER's rules predict for discrete classes so, prior to learning, the \eq{w1} 
results
for the 211 components  were sorted and divided into five classes (``\_1, \_2, \_3, \_4, \_5'')
with (roughly)
equal population size
(equal frequency discretization~\cite{dou95})- see \fig{b1}.
The quirks in the severity one data (discussed above) lead to the following
modeling decision: fuse together the top two classes;
i.e. class ``$\_12 = \_1 \cup \_2$''.

\begin{figure}
\begin{center}
\includegraphics[width=2.5in]{tim/score1.pdf}
\end{center}
\caption{Sorted values for \eq{w1} seen in 211
NASA software components. The y-axis
shows the breaks $b_1, b_2, b_3, b_4,b_5$
that divide the data into 5 classes.
Each class $x$ comprises the data
$b_{x} < x \le b_{x+1}$ or, for the top class,
$b_5 < x \le max$.
}\label{fig:b1}
\end{figure}

%\bi
%\item
%Once where the \eq{w} values were divided into five target classes: \_1, \_2, \_3, \_4, \_5
%representing the bins containing (left to right) the most to least high severity issues.
%\item
%Once where the \eq{w} values were divided into four target classes, generated as follows.
%After applying five-binning, the two bins representing severity 1 and severity 2 issues
%were  combined  into one. In the four class system, we denote 
%the combined class as
%$\_1'=\_1 \cup\_2$. 
%\ei
\begin{figure}
\begin{center}
\footnotesize
\begin{tabular}{rrlr}
& &&number of times \\
group & feature	&notes &selected\\\hline
1&us & use of standards & 10\\\hline
2&uc &config management   &        9\\
&ra & reuse approach    &      9\\
&am & artifact maturity     &     9\\\hline

3&fr &  formal reviews    &     8\\
&ex &experience        &   8\\\hline

4&ss& size of system &          7\\\hline

5&rm & risk management &          6\\\hline

6& cl & CMM level &         5\\
& dt & defect tracking  &       5\\
& do& development organization    &     4\\
& di& degree of innovation     &    4\\
&hs&human safety	    &   3\\
&as&   asset safety      &  2\\
&cx&  complexity        & 2\\
&pf& performance          &1
\end{tabular}
\end{center}
\caption{Number of times a feature was selected in 10-way
experiments where features were assessed using 90\% of
the data (selected at random).}\label{fig:fss}
\end{figure}

\begin{figure*}[!t]
\begin{center}
\scriptsize
\begin{tabular}{r|l@{}r|rrrr|rl@{~}r}
\multicolumn{3}{c}{~}&\multicolumn{4}{c}{$f$-measures (from \eq{F}) }& \\\cline{4-7}
treatment	&features	&\#features&\_12	&\_3	&\_4	&\_5	& \multicolumn{2}{l}{$F= \left(\sum f\right)/4$}&\\\cline{1-9}
    &               &  &    &       &      &        &    &                 &\\
A	&all - L1 - L2 - group(6)	&8&0.97	&0.95	&0.97	&0.99	&0.97&\rule{49mm}{2mm} &\\
B	&all  - L1 - L2 - group(5 + 6)	&7&0.95	&0.94	&0.97	&0.96	&0.96&\rule{48mm}{2mm}\\\
C	&all - L1 - L2 - group(4 + 5 + 6)	&6&0.93	&0.95	&0.98	&0.93	&0.95& \rule{47mm}{2mm}\\
D	&all - L1 - L2  	&16&0.94	&0.94	&0.93	&0.96	&0.94&\rule{47mm}{2mm}\\
E	&all - L1 - L2 - group(3 + 4 + 5 + 6)	&4&0.93	&0.97	&0.90	&0.87	&0.92& \rule{46mm}{2mm} &$\Rightarrow$ see \fig{rxE}\\
%F	&L1+ L2	&2&0.96	&0.89	&0.97	&0.70	&0.88&\rule{44mm}{2mm}\\
F	&co*ep, co + ep	&3&0.94	&0.84	&0.55	&0.70	&0.76&\rule{38mm}{2mm}\\
G	&L1	&1&0.67	&0.69	&0.00	&0.46	&0.45&\rule{22mm}{2mm}\\
H	&just "us"	&1&0.64	&0.60	&0.00	&0.00	&0.31&\rule{15mm}{2mm}\\
I	&L2	&1&0.57	&0.00	&0.32	&0.00	&0.22&\rule{11mm}{2mm}
\end{tabular}
\end{center}
\caption{Results from treatments. 
For a definition of the ``groups'' used in the second column, see \fig{fss}.}\label{fig:rx}
\end{figure*}


%\begin{figure*}[!t]
%\footnotesize
%\begin{center}
%\begin{tabular}{r|cr|rrrrr|rl}
%&&\multicolumn{1}{c}{~}&\multicolumn{5}{c}{$f$ measure for different classes}\\\cline{4-8}
%treatment&number of classes&features&\multicolumn{2}{l}{$\_1'=\_1\cup \_2$}&\_3&\_4&\_5&F&\\\hline
%4a&4&L2&0.57&&0.00&0.31&0.00&0.22&\rule{7mm}{2mm}\\
%4b&&us&0.64&&0.60&0.00&0.00&0.31&\rule{10mm}{2mm}\\
%4c&&L1&0.67&&0.69&0.00&0.46&0.46&\rule{15mm}{2mm}\\
%%4d&&l1+l2&0.96&0.89&0.94&0.68&&0.87&\rule{30mm}{2mm}\\
%4d&&all 16 raw SILAP1 features&0.93&&0.95&0.87&0.90&0.91&\rule{30mm}{2mm}\\
%4e&&us+am+ex+uc+fr+rm&0.95&&0.94&0.97&0.92&0.94&\rule{31mm}{2mm}\\
%4f&&ex+us+am&0.95&&0.96&0.955&0.94&0.95&\rule{32mm}{2mm}\\\hline
%\multicolumn{10}{c}{~}\\
%treatment&number of classes&features&\_1&\_2&\_3&\_4&\_5&F&\\\hline
%5a&5&L2&0.00&0.46&0.00&0.31&0.00&0.15&\rule{5mm}{2mm}\\
%5b&&L1&0.00&0.54&0.69&0.00&0.46&0.34&\rule{11mm}{2mm}\\
%5c&&us&1.00&0.58&0.60&0.00&0.00&0.44&\rule{15mm}{2mm}\\
%%5d&&l1+l2&1.00&0.94&0.86&0.94&0.72&0.89&\rule{30mm}{2mm}\\
%5d&&all 16 raw SILAP1 features&0.97&0.94&0.93&0.92&0.94&0.94&\rule{31mm}{2mm}\\
%5e&&us+ex+am+rm&0.97&0.95&0.96&0.94&0.93&0.95&\rule{32mm}{2mm}
%\end{tabular}
%\end{center}
%\caption{$f$ measures (from \eq{f})
%and $F$ measures (from \eq{F}) for different learning treatments.
%Note that the \_1' class of the four class system
%contains the union of all the \_1 and \_2 classes of the five class
%system. }\label{fig:ff}
%\end{figure*}
%
\subsection{Feature Pruning}

\fig{fss} shows the WRAPPER results
from
ten experiments using RIPPER as the target learner.
Each run used a randomly selected 90\% sample of the data.
This figure is divided into {\em groups}:
\bi
\item
Group 1 are those features that were {\em always} selected
in all ten pruning experiments. This group contains
only one feature: use of standards ($us$).
\item
Group 2 and 3 contain features that were 
{\em often} selected.
\item
Groups 4 and 5 contain features that were selected
in the majority of experiments.
\item
Group 6 are those features that were {\em not} selected
in the majority of the pruning experiments.
\ei
Group 6 raises some questions about the current version of SILAP.
For example,
the top five features of SILAP (measured
in terms of {\em contribution to goal}, see \fig{cons})
are
$hs$, $ex$, $pf$, $cx$, and $do$. All but one
of these (ex), 
appear in group 6 of \fig{fss}; i.e. are usually {\em not} selected by the WRAPPER PR.
That is,
empirically,
these features are far less influential
than suggested by SILAP.
This result motivated us to develop
an second version of SILAP.

To learn SILAP2, 
experiments were then conducted with RIPPER, using various feature subsets:
\bi
\item 
The subsets of selected features found in the groups of \fig{fss};
\item
All the {\em raw} SILAP features; 
\item
The project name ($L1$)
or the
component type ($L2$); 
\item
Combinations of the top-most derived SILAP model: $co$, $ep$, and $co*ep$.
\ei

\subsection{Performance Measures}

The learned rules were assessed using $F$; i.e. the  average f-measure
seen in all the target classes. 
$F$ is defined as follows.
Let $\{A_x,B_x,C_x,D_x\}$ denote the
true negatives, 
false negatives, 
false positives, and 
true positives
(respectively) for class $x$.
$Recall$ (or $pd$) is:
\begin{equation}
pd_x=recall_x=D_x/\left(B_x+D_x\right)
\end{equation}
Also  $Precision$ (or $prec$) is:
\begin{equation}
prec_x=precision_x=D_x/\left(D_x+C_x\right)
\end{equation}
The $f$-measure is the harmonic mean of precision and
recall. If {\em either} precision or recall
is low, then the $f$-measure is decreased. The $f$-measure is useful
for dual assessments that include {\em both} precision and recall.
\begin{equation}\label{eq:f}
f_x=(2 \cdot prec_x \cdot pd_x) / (prec_x + pd_x)
\end{equation}
Given $n$ target classes, then one measure of the total
performance is $F$, i.e. the average $f_x$ measure seen in  all classes:
\begin{equation}\label{eq:F}
F=\frac{1}{n}\sum f_x
\end{equation} 
$F$
ranges over $0\le F\le 1$ and {\em larger} $F$ values are {\em better}.

The $F$ measures were collected in
10-way  cross-validation
experiments. 
To conduct such an experiment,
the available data is divided into $N$ buckets.
For each bucket in a $N=10$-way cross-validation, a model is
learned on nine of the buckets, then tested on the remaining
bucket.  
Cross-validation
is the 
preferred method when testing
predictors intended for  predicting future 
events~\cite{witten05}.
%
%\begin{figure}
%\scriptsize
%\begin{alltt}
%     if (ss <= 2) and (uc >= 2)               => \_5  (29/0)
%else if (rm = 1) and (ra >= 2) and (ex = 1)   => \_5  (12/1)
%else if (ss = 1)                              => \_4  (16/0)
%else if (fr >= 2) and (us = 5)                => \_4  (17/0)
%else if (ex = 1) and (us >= 2)                => \_4  (11/2)
%else if (ex = 1)                              => \_3  (30/0)
%else if (fr >= 2) and (uc <= 2) and (ra <= 3) => \_3  (20/1)
%else 							              => \_12 (76/0)
%\end{alltt}
%\caption{$F=0.97$ rules from treatment A of \protect\fig{rx},
%rewritten in the simplified form of \tion{quirks}.
%}\label{fig:rxA}
%\end{figure}
%\begin{figure}
%\scriptsize
%\begin{alltt}
%(uc >= 2) and (us = 1)               => \_5 (12.0/0.0)
%(am = 3)                             => \_5 (19.0/3.0)
%(uc >= 2) and (am = 1) and (us <= 2) => \_5 (9.0/2.0)
%(am = 1) and (us = 2)                => \_4 (24.0/0.0)
%(us = 3) and (ra >= 4)               => \_4 (21.0/5.0)
%(us = 1)                             => \_3 (30.0/0.0)
%(ra = 3)                => \_3 (20.0/0.0)
%else                                  => \_12 (76.0/2.0)  
%\end{alltt}
%\caption{$F=0.92$ rules from treatment E of \protect\fig{rx}.
%rewritten in the simplified form of \tion{quirks}.
%}\label{fig:rxE}
%\end{figure}
%
%uc \in \{2,3\} or
%us \in \{1,2,3,5\} or
%am \in  \{1,2,3\} or
%ra \in \{1,4,5\}

%dont' know about uc \in \{4,5}
%or us =4
%or am \in \{4,5} pr
%ra=2
\subsection{Results}

\fig{rx} sorts the $F$ values found in 
10-way
cross-validation experiments on various
feature subsets using RIPPER.
The highest performing predictors used
just $\frac{8}{16}$ of the SILAP features (see Treatment A).
Also, 
nearly equivalent performance was found using
just $\frac{4}{16}$ of the features (see treatment E). 



One interesting aspect of \fig{rx} is that 
the derived features from the current
version of SILAP (Treatment F) performed relatively
poorly. This
result is evidence that the current version of SILAP needs modification.

Also, using just the project name ($L1$)  
performed poorly. If this were otherwise,
a project's name would be a good predictor for issue
severity. It this were generally the case, then
there would be no generality in NASA's issue reports.

Further, 
the component type ($L2$) proved to be a poor predictor of issue severity.
That is, there was no evidence from this data that any of
GNC, GROUND, CDH, or PAYLOAD are inherently more risky that any of the others.
%
%\begin{figure*}
%\scriptsize
%\begin{center}
%\begin{tabular}{r@{:~}l@{~}|r|@{~}r|@{~}r|@{~}r|@{~}r|@{~}|r@{~}|r@{~}|r@{~}|r@{~}|r@{~}||r|@{~}r|@{~}r|@{~}r|@{~}r|@{~}|r@{~}|r|@{~}r|@{~}r|@{~}r|}\cline{3-22}
%\multicolumn{2}{r|}{feature}&\multicolumn{5}{|c}{uc}&\multicolumn{5}{|c}{us}&\multicolumn{5}{|c}{am}&\multicolumn{5}{|c|}{ra}\\\cline{3-22}
%\multicolumn{2}{r|}{range}&1&2&3&4&5            &1&2&3&4&5             &1&2&3&4&5             &1&2&3&4&5\\\cline{3-22}
%\multicolumn{2}{r|}{frequency of range }         &116&77&18&0&0&50&84&58&0&19&104&88&19&0&0&91&4&39&48&33\\\cline{3-22}
%\multicolumn{22}{c}{~}\\\hline
%1&if $uc \ge 2 \wedge us =1$    then \_5     &&\_5&\_5&\ding{55}&\ding{55}&\_5&& &\ding{55}& &&&&\ding{55}&\ding{55}&&\ding{55}&&&\\
%2&else if $am = 3$ then \_5                    &&&&\ding{55}&\ding{55}&&&&\ding{55}&& & &\_5&\ding{55}&\ding{55}&&\ding{55}&&&\\
%3&else if $uc \ge 2 \wedge am=1 \wedge us \le 2$ then \_5&&\_5&\_5&\ding{55}&\ding{55}&\_5&\_5&&\ding{55}&&\_5&&\ding{55}&\ding{55}&\ding{55}&&\ding{55}&&&\\
%4&else if $am=1 \wedge us=2$  then \_4           &&&&\ding{55}&\ding{55}&&\_4&&\ding{55}&&\_4&&\ding{55}&\ding{55}&\ding{55}&&\ding{55}&&&\\
%5&else if $us=3 \wedge ra \ge 4$ then \_4          &&&4&\ding{55}&\ding{55}&&&\_4&\ding{55}&&&&\ding{55}&\ding{55}&\ding{55}&&\ding{55}&&\_4&\_4\\
%6&else if $us=1$ then \_3&\_3                  &&&\ding{55}&\ding{55}&&&&\ding{55}&&&&\ding{55}&\ding{55}&\ding{55}&&\ding{55}&&&\\
%7&else if $ra=3$ then \_3                      &\ding{55}&&&\ding{55}&\ding{55}&&&&\ding{55}&&&&\ding{55}&\ding{55}&\ding{55}&&\ding{55}&\_3& & \\
%8&else \_12                 &\ding{55}&\_12&\_12&\ding{55}&\ding{55}&\_12&\_12&\_12&\ding{55}&\_12&\_12&\_12&\ding{55}&\ding{55}&\ding{55}&\_12&\ding{55}&\ding{55}&\_12&\_12\\\hline
%\end{tabular}
%\end{center}
%\caption{Eight rules found by these study. Each rule tests for some range of the features
%(row three shows the frequency of each range).
%A ``\ding{55}'' denotes non-existent ranges; i.e. either rare or missing ranges
%or  
%ranges ruled out by higher rules.}\label{fig:rxE}
%\end{figure*}


\begin{figure}
\footnotesize
\begin{center}
\begin{tabular}{@{rule~}rrl@{~then ~}c}
1&if&  $uc \ge 2 \wedge us =1$    & \_5\\
2&else if&  $am = 3$ & \_5 \\
3&else if&  $uc \ge 2 \wedge am=1 \wedge us \le 2$ & \_5\\
4&else if&  $am=1 \wedge us=2$  & \_4    \\
5&else if&  $us=3 \wedge ra \ge 4$ & \_4  \\
6&else if&  $us=1$ & \_3        \\
7&else if&  $ra=3$ & \_3            \\
8&else if&  true &\_12              
\end{tabular}
\end{center}
\caption{The eight rules found by treatment E.}
\label{fig:rxE}
\end{figure}

Treatment E is very simple (uses just four features)
and performs nearly as well as anything else.
Hence, 
the rest of this article will focus on treatment E.

The rules
learned by treatment E using 100\% of the training
data are shown in \fig{rxE}.  
%In that figure,  missing ranges and {\em ruled-out} ranges
%are shown with  a ``\ding{55}''
%A range can be categorically {\em ruled out} when an earlier rule
%has a  condition that tests only
%that range.
%For example, row 2  tests for $am=3$ and predicts for issues of severity ``\_5''. Hence, rule 2 rules out $am=3$ from all subsequent rules.
%Note also that these rules are silent on ranges that are missing,
%or only occur rarely,  in the training
%data; i.e. 
%$uc\in \{4,5\}$ or $us \in \{4\}$ or $am \in \{4,5\}$ and
%$ra \in \{2\}$. The final row  of \fig{rxE} shows the ranges associated
%with  the most
%issues of highest severity. 
%
SILAP2
replaces
the error potential calculations
of \fig{SILAP.awk} with \fig{rxE}. The issue severity classes
\_5, \_4, \_3, \_12 
 can be mapped to error potential 1, 2, 3, 4 or 5
of \fig{silapModel} (respectively). 

SILAP2 does not change the
{\em consequence} calculation of \fig{SILAP.awk}. 
 Human safety ($hs$) should always be more important that
asset safety ($as$)
or performance issues ($pf$) that compromise system goals.
Otherwise, {\em consequence} can be modeled as a linear combination of $as$ and $pf$.

\section{Task Pruning}

\PARstart{F}{igure} \ref{fig:silapModel} can use SILAP2's
calculation of  {\em error potential} and SILAP's calculation
of $consequence$ to {\em select}
 a list of IV\&V
tasks. In two situations, that list should be {\em pruned}:
\bi
\item
If the tasks costs
more than
the available budget.
\item
If SILAP2's combining of \_1 and \_2 into \_12 means that too
many tasks are selected.
\ei
This last situation can occur 
with 
eight tasks of \fig{silapModel}.
With tasks 2.4, 2.5, 2.6, 4.7, 5.6, 5.8, 5.10 and 6.1,
the tasks selected by {\em error potential=5} would not be selected when {\em error potential=4}.
This is not an ideal situation
but,  our current data has too few examples of severity 1 issues
to support the generation of good
level \_1 predictors.


%\begin{figure*}
%\scriptsize
%\begin{center}
%\begin{tabular}{r|r|r|r@{\%~}l|r|l}
%type & $\Delta$ & rank     & \multicolumn{2}{l|}{normalized 
%$\left(\frac{Issues\cdot 10^{(5-Severity)} \cdot \Delta^{(6-Phase)}}{Costs \cdot Frequency}\right)$ } &Task id & Task\\\hline
%robot&5&   1   &    52& \rule{52mm}{2mm}& 2.5& SW/User Requirements Allocation Analysis\\ 
% &&    2   &    22& \rule{22mm}{2mm}& 2.6&                    Traceability Analysis\\ 
%  &&   3   &     9&  \rule{9mm}{2mm}& 2.4&              Concept Document Evaluation\\ 
%   &&  4   &     4&  \rule{4mm}{2mm}& 3.4&                System Test Plan Analysis\\ 
% &&    5   &     4&  \rule{4mm}{2mm}& 3.3&        Interface Analysis - Requirements\\ 
%  &&   6   &     3&  \rule{3mm}{2mm}& 3.1&    Traceability Analysis -  Requirements\\ 
%&&     7   &     2&  \rule{2mm}{2mm}& 3.2&         Software Requirements Evaluation\\ 
%&&     8   &     2&  \rule{2mm}{2mm}& 2.2&         Software Architecture Assessment\\ 
%&&     9   &     1&  \rule{1mm}{2mm}& 4.6&                        Database Analysis\\ 
%&&    10   &     1&  \rule{1mm}{2mm}& 2.3&               System Requirements Review\\ 
%&&    11   &     1&  \rule{1mm}{2mm}& 2.1&                          Reuse Analysis*\\\hline 
%
%robot&2&     1   &    39& \rule{39mm}{2mm}& 2.5& SW/User Requirements Allocation Analysis\\
%&&     2   &    16& \rule{16mm}{2mm}& 2.6&                    Traceability Analysis\\ 
%&&     3   &     8&  \rule{8mm}{2mm}& 3.4&                System Test Plan Analysis\\ 
%&&     4   &     8&  \rule{8mm}{2mm}& 3.3&        Interface Analysis - Requirements\\ 
%&&     5   &     7&  \rule{7mm}{2mm}& 2.4&              Concept Document Evaluation\\ 
%&&     6   &     5&  \rule{5mm}{2mm}& 4.6&                        Database Analysis\\ 
%&&     7   &     5&  \rule{5mm}{2mm}& 3.1&    Traceability Analysis -  Requirements\\ 
%&&     8   &     3&  \rule{3mm}{2mm}& 3.2&         Software Requirements Evaluation\\ 
%&&     9   &     2&  \rule{2mm}{2mm}& 5.4&                System Test Case Analysis\\ 
%&&    10   &     1&  \rule{1mm}{2mm}& 6.4&             System Test Results Analysis\\ 
%&&    11   &     1&  \rule{1mm}{2mm}& 5.5&               Software FQT Case Analysis\\ 
%&&    12   &     1&  \rule{1mm}{2mm}& 4.2&               Software Design Evaluation\\ 
%&&    13   &     1&  \rule{1mm}{2mm}& 2.3&               System Requirements Review\\ 
%&&    14   &     1&  \rule{1mm}{2mm}& 2.2&         Software Architecture Assessment\\ 
%&&    15   &     1&  \rule{1mm}{2mm}& 2.1&                          Reuse Analysis*\\ \hline
%
%human&5&     1   &    59& \rule{59mm}{2mm}& 2.5& SW/User Requirements Allocation Analysis\\ 
%&&     2   &    24& \rule{24mm}{2mm}& 2.6&                    Traceability Analysis\\ 
%&&     3   &    10& \rule{10mm}{2mm}& 2.4&              Concept Document Evaluation\\ 
%&&     4   &     2&  \rule{2mm}{2mm}& 2.2&         Software Architecture Assessment\\ 
%&&     5   &     1&  \rule{1mm}{2mm}& 3.4&                System Test Plan Analysis\\ 
%&&     6   &     1&  \rule{1mm}{2mm}& 3.3&        Interface Analysis - Requirements\\ 
%&&     7   &     1&  \rule{1mm}{2mm}& 2.3&               System Requirements Review\\ 
%&&     8   &     1&  \rule{1mm}{2mm}& 2.1&                          Reuse Analysis*\\\hline 
%
%
%human&2&     1   &    56& \rule{56mm}{2mm}& 2.5& SW/User Requirements Allocation Analysis\\ 
%&&     2   &    23& \rule{23mm}{2mm}& 2.6&                    Traceability Analysis\\ 
%&&     3   &    10& \rule{10mm}{2mm}& 2.4&              Concept Document Evaluation\\ 
%&&     4   &     2&  \rule{2mm}{2mm}& 3.4&                System Test Plan Analysis\\ 
%&&     5   &     2&  \rule{2mm}{2mm}& 3.3&        Interface Analysis - Requirements\\ 
%&&     6   &     2&  \rule{2mm}{2mm}& 2.2&         Software Architecture Assessment\\ 
%&&     7   &     1&  \rule{1mm}{2mm}& 5.4&                System Test Case Analysis\\ 
%&&     8   &     1&  \rule{1mm}{2mm}& 4.6&                        Database Analysis\\ 
%&&     9   &     1&  \rule{1mm}{2mm}& 3.2&         Software Requirements Evaluation\\ 
% &&   10   &     1&  \rule{1mm}{2mm}& 3.1&    Traceability Analysis -  Requirements\\ 
% &&   11   &     1&  \rule{1mm}{2mm}& 2.3&               System Requirements Review\\ 
%&&    12   &     1&  \rule{1mm}{2mm}& 2.1&                          Reuse Analysis* 
%\end{tabular}
%\end{center}
%\caption{Value of \fig{silapModel} tasks; values normalized to sum to 100.}\label{fig:value}
%\end{figure*}

To support pruning, \fig{allocated} sorts the IV\&V tasks according to {\em phase effectiveness},
then {\em effort expended}. 
These sorts are explained below. Note that, if pruning is required, tasks lower in the sort should be pruned first.

{\em Phase effectiveness} was calculated using data from the
9 robotic missions of
\fig{issues}.
The data from the human-rated missions was ignored since
launching humans into space is a highly specialized activity that may not be relevant
to the broader 
field of software engineering.


\begin{figure}[!t]
\scriptsize
\begin{center}
\begin{tabular}{r|r|r|r|r|r}
sort order: \\
best=1\\
worst=27&$p$=phase & ~task&  ~cost &~frequency&frequency*cost\\\hline
1&3. requirements   &         3.5&         0.77&5&4\\
2&  &          3.4&         1.62&50&81\\
3& &           3.3&         1.6&53&85\\
4&&            3.1&         1.46&90&131\\
5&&            3.2&         3.89&54&210\\ \cline{5-6}
& &              &             & sub-total&$\alpha_3=512$\\\hline
6&2. concept &            2.5&         0.15&13&2\\
7& &           2.6&         0.36&13&5\\
8&      &      2.4&         0.86&13&11\\
9&    &        2.2&         1.28&50&63\\
10&     &       2.3&         1.67& 54&90\\
11&&            2.1&         1.53&71&109\\ \cline{5-6}
& &              &             &  sub-total&$\alpha_2=281$\\\hline
12&         4.  design &      4.5&         1.43&5&7\\
13&       &     4.6&         0.63&35&22\\
14&    &        4.3&         2.50&9&23\\
15&   &         4.2&         4.31&31&134\\
16&& 4.1&         2.71&81&220\\
17&     &       4.4&         2.84&88&250\\ \cline{5-6}
 & &              &             &  sub-total&$\alpha_4=655$\\\hline
18&5. implement &        5.11&         1.82&3&5\\
19&   &         5.4&         1.16&26&30\\
20&  &          5.5&         2.34&26&61\\
21&  &          5.3&         5.06&53&268\\
22&&            5.1&         3.49&81&283\\
23&        &    5.2&         10.28&53&545\\ \cline{5-6}
 & &              &             & sub-total&$\alpha_5=1192$\\\hline
24&6. test &    6.4&         2.08&26&54\\
25&       &     6.3&        20.93&3&63\\
26&         &   6.5&         3.20&26&83\\
27&&            6.1&         2.30&88&202\\ \cline{5-6}
 & &              &             & sub-total&$\alpha_6=402$\\\hline
\end{tabular}
\end{center}

\caption{Sort order of IV\&V tasks: top to bottom, best to worst.
Tasks are only listed if {\em both} ``cost to apply task'' {\em and} 
``frequency 
of applying that task'' data is available.
Costs  frequency data is
based on historical
records at NASA IV\&V. For proprietary  reasons, cost numbers
are
normalized to 100. $\alpha_p$ is the sum of the $frequency*cost$ figures
for each phase.
}\label{fig:allocated}
\end{figure}

%\begin{figure}[!b]
%\begin{center}
%\scriptsize
%\begin{tabular}{r@{~}l|r@{~}l|r@{~}l|r@{~}l|r@{~}l}
% &              & \multicolumn{8}{c}{$\frac{issues'}{cost*frequency} = \frac{\beta_p
%\cdot  \Delta^{(6-p)}}{\alpha_p}$ }\\\cline{3-10}
%&               & \multicolumn{4}{c|}{robotic} & \multicolumn{4}{c}{manned}\\\hline
%$p =$&phase   &    $\Delta=2$ & &$\Delta=5$  & &$\Delta=2$ & &$\Delta=5$&\\\hline
%2&concept     &    713 (18\%) & \rule{4mm}{2mm}  &  27,867 (39\%) & \rule{10mm}{2mm} & 39,216  (42\%) & \rule{10mm}{2mm}& 1,531,876  (66\%) & \rule{16mm}{2mm} \\
%3&requirements&  2,688 (67\%) & \rule{17mm}{2mm} &  41,993 (58\%) & \rule{15mm}{2mm} & 51,067  (55\%) & \rule{13mm}{2mm}&   797,924  (34\%) & \rule{8mm}{2mm}  \\
%4&design      &    286 ~(7\%) & \rule{2mm}{2mm}  &   1,788 ~(2\%) &                  &    828  ~(1\%) &                 &     5,178  ~(0\%) &                  \\
%5&implement   &    165 ~(4\%) & \rule{1mm}{2mm}  &     412 ~(1\%) &                  &    950  ~(1\%) &                 &     2,376  ~(0\%) &                  \\
%6&test        &    146 ~(4\%) & \rule{1mm}{2mm}  &     146 ~(0\%) &                  &    459  ~(0\%) &                 &       459  ~(0\%) &             \\\hline
%
% &    totals      & $\sum=3,998$   &                 & $\sum=72,206$    &  &   $\sum=92,521$ && $\sum=2,337,813$   &
%\end{tabular}
%\end{center}
%\caption{Heuristic assessment of the relative merits of IV\&V tasks at different phases.
%The values of $\alpha_p$ and $beta_p$ come from \fig{allocated}
%and \fig{issues}. 
%}\label{fig:phase}
%\end{figure}
%

\begin{figure}
\scriptsize
\begin{center}
\begin{tabular}{r@{~}l|@{~}r@{~}l|@{~}r@{~}l}
% &            &\multicolumn{4}{r}{ 
%$\frac{issues'}{cost*frequency} = \frac{\beta_p \cdot  \Delta^{(6-p)}}{\alpha_p}$ }\\\cline{3-6}

$p$&phase   &    $\gamma(\Delta=2)$ &                  &$\gamma(\Delta=5)$      & \\\hline
2&concept     &    713 :18\% & \rule{4mm}{2mm}  &  27,867 :39\% & \rule{10mm}{2mm}  \\
3&requirements&  2,688 :67\% & \rule{17mm}{2mm} &  41,993 :58\% & \rule{15mm}{2mm}   \\
4&design      &    286 ~:7\% & \rule{2mm}{2mm}  &   1,788 ~:2\% &       \\
5&implement   &    182 ~:4\% & \rule{1mm}{2mm}  &     454 ~:1\% &         \\
6&test        &    146 ~:4\% & \rule{1mm}{2mm}  &     146 ~:0\% &          \\\hline

 &    totals      & \multicolumn{2}{c}{$\sum\gamma=4,015$}   
                 & \multicolumn{2}{c}{$\sum\gamma=72,248$}
\end{tabular}
\end{center}
\caption{Heuristic assessment of the relative merits of IV\&V tasks at different phases.
The $\alpha_p$, $beta_p$ and $\Delta$ values
come from
and \fig{allocated}, \fig{issues},  and  \fig{dabney06a}.
}\label{fig:phase}
\end{figure}

An ideal set of IV\&V tasks finds many issues, early in the life cycle, at low cost, after
performing just a few tasks. This could be modeled as:
\begin{equation}\label{eq:bg}
\gamma(\Delta) = \frac{issues\; found\; early}{effort\; expended}
=\frac{\beta_p
\cdot  \Delta^{(6-p)}}{\alpha_p} 
\end{equation}
In this expression, $\gamma$ is {\em phase effectiveness} and
\bi
\item[$\alpha_p$]:  (from \fig{allocated}) is total effort expended per phase;
\item[$\beta_p$]: (from \fig{issues})
 sums the issues found in phase $p$ times a factor that weights high severity
issues exponentially larger than low severity issues; and  
\item[$\Delta^{6-p}$]:  (from \fig{dabney06a}) rewards issues found earlier in the life cycle in phase $p$. 
\ei
\fig{phase} shows the $\gamma$ calculated from our data.
Each column shows the \eq{bg} values as a percentage of the totals
for that column.
The phases of \fig{allocated} are sorted according to their
$\gamma$ values.
To obtain a more fine-grain ordering, \fig{allocated} sorts
tasks within a
phase by effort expended; i.e. $\alpha$.

Note that, according to \fig{allocated}, 
most of the cost-benefit of IV\&V is seen
in the early life cycle phases of
requirements (first) and concept phase (second).
Later life cycle IV\&V tasks
add very little to IV\&V effectiveness
(in \fig{allocated}, only 3\% to 15\%).
This informs the problem of the eight cases where the rules
of SILAP2 can over-specify the IV\&V tasks. 
Note
that five of those eight cases are late life cycle tasks; i.e. can be pruned without degrading much of the value
of IV\&V. 

As to the remaining three problem cases 
(the concept phase tasks of 2.4, 2.5, 2.6),
as shown by the ``Z'' notation in \fig{silapModel},
these tasks are only for NASA-specific
human-rated space missions. That is, software engineers
using SILAP2 outside of NASA will never see task over-specification
on tasks 2.4,2.5, and 2.6.
And even if they do, the net effect many be minimal.
\fig{allocated} lists these tasks
 as having the smallest effort expended values of
any of the concept phase IV\&V tasks.
Hence, if they are selected inappropriately,
these tasks will add little to the total IV\&V cost. 

\begin{figure}[!t]
\begin{center}
\scriptsize
\begin{tabular}{r@{~}r@{~}l}
requirements & 53\% & \rule{53mm}{2mm}\\
design & 27\% & \rule{27mm}{2mm}\\
integration \& test & 14\% & \rule{14mm}{2mm}\\
engineering specialties & 4\% & \rule{4mm}{2mm}\\
coding \& unit test & 2\% & \rule{2mm}{2mm}\\
\end{tabular}
\end{center}
\caption{Patterns of risk within product engineering; from \protect\cite{hig96}.}\label{fig:huguera}.
\end{figure}
As an aside, we note that the 
sort ordering of phases
in \fig{allocated} is not without precedent.  Writing over a decade
ago, Higuera and Haimes~\cite{hig96} reviewed classes of risk
associated with the Software Engineering Institute's risk taxonomy.
They concluded that the risks associated with product engineering,
development environments, and the general development program were
roughly evenly distributed (30\%, 33\%, 37\%).  However, \fig{huguera}
shows the risk within  product engineering (i.e. the region usually
studied by IV\&V).  Note that their risk distribution matches the
general pattern of \fig{phase}: late life cycle activities offer
very little to the overall risk.
%\begin{figure}
%\begin{center}
%\includegraphics[width=2in]{allcosts.pdf}
%\end{center}
%\caption{All task costs from \fig{allocated}, sorted.}\label{fig:allcosts}
%\end{figure}
\subsection{Alternate Sorting Orders}\label{sec:except}

In certain circumstances, the above pruning criteria should be changed. 
Firstly, for mission and safety-critical projects, pruning IV\&V tasks
is {\em not} recommended (to say the least).

Secondly, different organizations have different cost structures and,
hence, different $\alpha_p$
values.  If such data is available, then 
\fig{allocated} could be sorted differently.

Thirdly, the current version of SILAP offers some coarse-grain pruning
heuristics that are different to \fig{allocated}.
For example, to achieve {\em slight} pruning,
all projects have n consequence $co$ score of one {\em unless}
$pf >2 \vee as > 3$ (in which case,
the $co$ calculation of \fig{SILAP.awk} is applied). SILAP2's
 pruning order (in \fig{allocated}) is more fine-grained that the
coarse-grain
pruning of the current SILAP.


\section{External Validity}

\subsection{Evaluation Bias}

\PARstart{T}{he} rules of SILAP2 were
based on the issue/severity summation of $W1_j$ seen in \eq{w1}.
To check the external validity of that sum, 
we explored the effects of a very different calculation.
Recall that according to $W1_j$ ,
issue reports are {\em thresholds} that must be reached
in order to {\em trigger} different activities.
An alternate interpretation,
called $W2_j$, is that
many lower level severities can become
more important than higher level severities. In this interpretation, issue reports are {\em votes}
that can {\em accumulate}.  $W2_j$ can be implemented as follows:
\begin{equation}\label{eq:w2}
W2_j= 1 + \sum_{i=1}^5 s_{i,j}\cdot10^{6-i}
\end{equation}


The 211 $W2_j$ scores generated from our data are shown in
\fig{b2}.  After dividing the scores
into 
five approximately equal buckets, 
five new classes are generated: \_1', \_2', \_3', \_4', \_5' representing
least to most severe issues (respectively).

The {\em confusion matrix} of \fig{confuse} compares the classes
generated from \fig{b2} and those found in \fig{b1} (from  \eq{w2}):
off-diagonal entries denote when the same class gets  assigned
different classes by \eq{w1} and \eq{w2}. Note that, except for
class ``\_1'' and ``\_2'', the classifications are nearly identical.
Since the severity one issue data is ``quirky'' (see above), this study
combined the class ``\_1'' and ``\_2'' into their union \mbox{``$\_12=
\_1 \cup \_2$''}.  Consequently, the region of confusion between
\eq{w1} and \eq{w2} disappears and the rules shown above are valid
across  a {\em range} of evaluation biases (from \eq{w1} to \eq{w2}).

\begin{figure}
\begin{center}
\includegraphics[width=2.5in]{tim/score1.pdf}
\end{center}
\caption{Sorted values for \eq{w2} seen in the 211
NASA software components. The y-axis
shows the breaks $b_1, b_2, b_3, b_4,b_5$
that divide the data into 5 classes.
}\label{fig:b2}
\end{figure}
\begin{figure}
\footnotesize
\begin{center}
\begin{tabular}{rr|rrrrr}
\multicolumn{2}{c}{~}&
\multicolumn{5}{c}{\underline{$W2$} }\\
\multicolumn{2}{c|}{~}&
          \_1'&  \_2' &   \_3'&   \_4'   &   \_5'\\\hline
 & \_1 & 19 & 12 &    &       &   \\
         & \_2 & 27 & 20 &     &      &    \\
\underline{$W1$}         & \_3 &    &    &  50 &   1  &     \\
         & \_4 &    &    &     &   42 &     \\
         & \_5 &    &    &     &      &   40\\
\end{tabular}
\end{center}
\caption{Comparison of 
classifications offered by \eq{w1} and \eq{w2} from \fig{b1} and \fig{b2}.
Each cell $(x,y)$
counts how many components have $class(W1)=x$ and
$class(W2)=y$.
}\label{fig:confuse}
\end{figure}

\subsection{Language and Search Bias}

All learners have a {\em language} and {\em search} bias.
For example, linear regression
often searches the Euclidean distance of actual to predicted values.
The learned theory can only be reported in the language of a linear model; i.e.
\mbox{$y=K+\sum c_ix_i$}.
Other learners search information theoretic measures of the data and
their learned models are only reported in the language of
decision trees or rules (e.g. RIPPER). Other
learners have other search and language biases and will
yield different models.

We must therefore defend the choice of learner used in this experiment.
The case for RIPPER and WRAPPER was made above: our results are based on data not collected
for the purposes of this study. Such data may contain signals not connected to our target function (\eq{w1}).
Hence, tools are required like RIPPER and WRAPPER that  remove superfluous parts of the data or learned model.

As to other kinds of data miners,
decision tree learners such as C4.5~\cite{quinlan92}
or CART~\cite{breiman84}
execute in local top-down
search, with no memory between different branches.  Hence, the same concept
can be needlessly repeated many times within the tree.
Consequently, such trees can be  needlessly complex.

Also, linear regression may not be indicated in this domain since there is some evidence of
non-linearity in our data. 
Consider \fig{rxE} and the situation where rules 1 \ldots 6 have failed.
In that case, the prediction falls to rules 7 and 8. If we change $ra$ from 2 to 3 to 4,
then the severity prediction changes in a non-linear manner from
 \_12, to \_3, then back to \_12 again. 
Such an effect cannot be modeled using the single linear model generated by standard linear
regression.

Similarly, we would caution against learners that learn models containing a single feature;
e.g. a study that reports that feature X has the strongest correlation to \eq{w1} and is
therefore {\em the} most important indicator of  issue severity.
In 
all these experiments
(\fig{fss} and \fig{rx})
any single feature performed worse than those that used
combinations of features. 
This is an encouraging result since, otherwise, experiments with
data miners that explore combinations of features (like RIPPER)
would be superfluous. 

There are more data mining methods besides RIPPER, WRAPPER, C4.5, CART, and linear regression.
The improvement offered by those learners could only ever be 
$0.92 \le F \le 1$ (i.e. above the performance of treatment E).
Hence, we are not motivated to move from
RIPPER+WRAPPER.

\subsection{Sampling Bias}

Apart from {\em language} and {\em search} bias, the results of any data mining
experiment suffer from  a {\em sampling} bias; i.e. the conclusions may be
local to the data used in training and not be relevant to other
populations.
For example:
\bi
\item 
All our conclusions were drawn from 211 NASA components and, as detailed above,
that data has numerous quirks: i.e.
missing ranges for
$uc\in \{4,5\}$, $us \in \{4\}$,  $am \in \{4,5\}$,
$ra \in \{2\}$. 
SILAP2 should be used with caution in projects that fall into these missing
ranges.
\item
This study reports that IV\&V tasks performed in the concept phase
are {\em less} useful than in the requirements phase. 
This result may be a result of current NASA development practices where more IV\&V
effort is devoted to requirements that concept analysis. 
\item
The data used here comes from
NASA and NASA
works in a particularly unique market niche:
human-rated missions to outer space. 
To minimize this concern, 
data from human-rated missions was not used
to learn the rules of \fig{rxE} or the sort order of \fig{allocated}.
\ei
Basili, Zelkowitz, et.al.~\cite{basili02}
have argued that conclusions from NASA data are relevant to the
general software engineering industry.
NASA makes extensive use of contractors
and these
contractors service many other industries. Such contractors are
contractually obliged (ISO-9001) to demonstrate their understanding
and usage of current industrial best practices.  For these reasons,
we believe that the IV\&V tasks proposed by SILAP2 would have
general applicability across the industry.
As evidence that this claim, we note that NASA
did not invent \fig{silapModel} from scratch. Rather, it adapted it
from IEEE standard V\&V practices (one of us, Costello, serves on the
committee that defines and extends that standard~\cite{1012}).

%\section{Tuning SILAP2 to Local Data}\label{sec:tune}
%
%\PARstart{T}{he} case for the external validity of 
%our conclusions was made above.
%Nevertheless,
%it is always useful to improve general models with local data.
%For example, 
%researchers in effort estimation report the efficacy of general cost models (e.g.
%COCOMO) can be improved via calibration to local data~\cite{boehm00b}.
%One such tuning was discussed above: if an organization has local $\alpha_p$
%data, they could re-sort to \fig{allocated}.
%
%Another tuning method would be to collect fresh issue data
%and SILAP features on local data. The effort involved with this  fresh data
%collection can be reduced in two ways.
%Firstly,  {\em many} components within a project can be built
%by the {\em same} contractor.
%In this case, some of the SILAP features can be collected once per
%contractor and quickly applied to multiple components.
%The {\em component-specific} features
%   and $hs$, $as$, $pf$, $cx$, $di$ and $ss$. The
%remaining features are typically {\em common} amongst all components
%generated by one contractor.  
%
%Secondly, 
%in the {\em optimistic} case, only the four features
%selected  by WRAPPER need to be collected. None of these 
%features are {\em component-specific}; i.e. they can be collected
%one per contractor than quickly copied across many components.
%
%(In the alternate {\em pessimistic} case,
%these four features do not yield issue predictors with large $F$ values
%and {\em all} the features must be collected.
%Data mining should be used to check if a particular domain
%falls into the {\em optimistic} or {\em pessimistic case}.)
%
%Hence, reasoning {\em optimistically}, we can generate an estimate
%for collecting the SILAP features.
% In {\em current practice}, it takes two weeks of
%   full-time work by a team of about five people to collect
%values for all the SILAP features;
%i.e. 
%analyze a system
%   and determine the values of the raw component scores of \fig{silapFeatures}.
%    In that ten day period (sometimes larger for more complex
%   projects) the project is divided among the members of the
%   criticality analysis team, each team member then reviews project
%   documentation and determines the raw scores along with a rationale
%   for that score.  The team then meets to compare scores.
%   reach a consensus on each
%   component, and document their conclusions in a report.
%
%Reasoning optimistically, an adjustment to the effort of current
%practice can be calculated as follows:
%\bi
%\item
%In the optimistic case, the size of the SILAP data collection team
%can be reduced from five to three people.
%\item
% We are collecting data on 5 projects with
%20 components 
%and an   average of 3 contractors each;
%i.e. we only need to collect 5*3=15  sets of SILAP features.
%\item
%The effort distribution for SILAP data collection
%is 
%   67\% for setting the component-specific features and 33\% for the 
%common features (these numbers come from our historical records).
%\item
%80\% of current practice is spent collecting the features,
%and the remaining 20\% is spent
%on
%    logistics (acquiring documents etc) and
%   report writing.  
%\ei
%Under these assumptions, 
%the effort required to collect fresh data would be
%{\em 15*3 analysts*10 days*(20\% + 80\% * 0.33)};
%i.e.   208 days. 
%
%

\section{Conclusions}
Early
detection of project issues can
significantly reduce the cost-to-fix defects.
Numerous prior studies have reported the effectiveness of such external
IV\&V teams at finding
project issues early in the life cycle~\cite{boehm88,dabney07,shu02,wallace89,arthur99}, 

Nearly 100 (I)V\&V tasks are known  in the literature~\cite{wallace89,1012,costello05}.
However, with respect to the cost-effectiveness
of those tasks, the literature is incomplete. 
We can only assess cost and effectiveness data on 27 of these tasks (see \fig{allocated}). Further,
except in certain circumstances (see \tion{except}),
we can only recommend eleven of those tasks for the purposes of IV\&V.
Perhaps  it is time  to spend less effort on the creation
of {\em new} tasks and more effort on the assessment of {\em existing} tasks.

In order to assess the cost-effectiveness of IV\&V tasks,
we applied data miners to historical logs from NASA IV\&V to
find selection and ranking rules for IV\&V tasks.
Due to large-scale on-going organizational changes within NASA,
the data available to this study was potentially noisy; i.e. contains 
signals not  necessarily connected to the target of predicting
for issue severity and frequency. Despite this, our data
mining found rules (see \fig{rxE})
with very high precision and recall 
values (see \fig{rx})
for predicting for different classes of issue
frequency and severity ($F$ values close to one, in a 10-way
cross-validation).  This excellent performance persists 
across a range of summarization methods for issues and
severities (from \eq{w1} to \eq{w2}). Other data miners
{\em might} do better than on this data but that
improvement could only ever be slight ($0.92 \le F \le 1$).

We attribute the success of the data mining to some careful
choices regarding modeling and learning:
\bi
\item
WRAPPER~\cite{kohavi97} removes unnecessary or noisy
features. 
\item
RIPPER~\cite{cohen95r} also handles noise by aggressively
pruning spurious parts of a learned model. 
\item
High $F$ values were achieved with WRAPPER+RIPPER {\em after} combining
the severity one and two issue reports. This combination
was necessary due to  quirks in the relatively infrequent severity one issue reports in our data.
\ei

 The RIPPER results comment on the merits of using linear models
 on these domain. The rules learned by RIPPER exhibit some
 non-linearities; i.e. standard modeling methods like
  linear regression would not be appropriate in this domain.

The WRAPPER results  comment on SILAP, the current
NASA IV\&V task selection model. Comparing \fig{cons} and
 \fig{fss}, it is clear that many of the features in the current
 version of SILAP are not useful for predicting issue and severity.
 Hence, a new version of SILAP was created. SILAP2 (in \fig{silap})
uses the consequence calculations of the current version
of SILAP, and replaces
the old error potential calculations with the rules learned
from WRAPPER+RIPPER.
SILAP2 uses only a quarter of the features of  SILAP
and, hence, is
faster to apply.

When SILAP2 selects too many tasks,
the selected tasks can be {\em pruned} 
(but note that IV\&V task pruning is {\em not}
recommend for mission-critical or safety-critical applications).
The current version of SILAP has some coarse-grained pruing rules while
SILAP2 offers 
a more detailed pruning criteria,
using historical logs of expended effort and
effectiveness (see \fig{allocated}). 

The analysis of this report is based on public domain data and software.
To the best of our knowledge, this is the first  {\em reproducible}
report
of an demonstrably useful issue frequency and
 severity predictor
that can be quickly applied early in a project's
life cycle.
Reproducibility
is an important pragmatic and methodological principle:
\bi
\item
 Pragmatically, it allows industrial
practitioners to tune our model to their local data regarding IV\&V task cost
effectiveness (if such data was available).
\item
Methodologically, it
allows researchers to confirm,
refute, or even improve prior results. 
\ei
In our view, in the field of software engineering,
there are all too few examples of reproduced, and extended, results\footnote{
Exception: see  the reports of the PROMISE
workshop
\url{http://promisedata.org/repository/papers.html}}. 
We hope that this report encourages more reproducible experiments
in software engineering.

Finally, this analysis has used historical records of previously
found issues to rank the IV\&V tasks used to find them.
A broader concern is what percentage of issues are missed by {\em all} 
IV\&V tasks. 
Unfortunately,  we have no data on this important concern.
The goal of complete software assurance is illusive; e.g. the state
space explosion problem prevents model checkers from rigorously checking
all but small mission critical portions of a system~\cite{me00y}. The best we can
do {\em currently} is to rank sets of candidate tasks according to their past
cost-effectiveness. In the future, we  can certainly
do better than this- but only after corporations
recognize the value in keeping consistent records, across multiple projects, across
many years. 


%
%\subsection{NASA-specific Conclusions }
%
%\subsubsection{Revise SILAP}
%
%SILAP needs some changes.
%\fig{epcow} shows that
%SILAP's calculated consequence and error potential values are poor predictors for
%issue severity. 
%
%XXX retain consequence
%
%\subsubsection{Adopt SILAP2}
%
%While the derived features of SILAP should be removed,
%the raw features of SILAP are demonstrably useful. As seen in \fig{ff},
%when processed by the WRAPPER
%and RIPPER, these raw features can be combined into an issue severity
%predictor with a near perfect $F$ measure.
%
%Our proposed SILAP2 model retains (a)~the task selection table of \fig{silapModel};
%(b)~the raw SILAP features of \fig{silapFeatures}; and (c)~the consequence of error (co)
%calculations of \fig{SILAP.awk}. However, it replaces the error potential calculation
%with the rules of \fig{4f}.
%
%The following
%syntactic redefinition is required to map the output of the rules to
%the task selection table:
%\bi
%\item
%Error potentials of 1,2,3,4,5 (in \fig{silapModel}) correspond to issue severity predictions
%(from \fig{ff}) of $\_5,\_4,\_3,\_1',\_1'$, respectively.
%\ei
%
%
%\subsubsection{Revise Current IV\&V Task Assignments}
%
%Since SILAP underestimates project risks, and SILAP2 is exhibiting good performance,
%all current IV\&V task assignments should be reassessed using SILAP2.
%
%
%\subsubsection{Revise NASA's Data Collection Procedures}
%
%The hundreds of  records found by this study were useful and informative. However, for a national
%software development organization, the goal should be thousands, or tens of thousands, of records. 
%Also, the performance of the SILAP raw features in SILAP2 
%suggests that when NASA IV\&V next
%changes 
%its databases it should, at the very least, retain the raw SILAP features.
%
%\subsection{General Conclusions}
%
%\subsubsection{Coping with Real World Data}
%
%Ideally, organizations are stable enough to develop a central reference model for their
%software development. Such a reference model is very useful for defining and interpreting
%data collection  and maturing a ``lessons learned'' knowledge base. Such an ideal situation existed
%with the  NASA SEL from 1983 to 1995. 
%
%Given the current pace of change of NASA 
%it is unrealistic to deploy the learning organization framework within NASA. Nevertheless,
%as shown here, heuristically gathered data can still be used to generate a knowledge base
%with a surprisingly high effectiveness ($F=0.95$ in a 10-way cross-validation).
%
%We attribute the high performance of our learners to the methods used
%to remove the noise present in most real world SE data sets. 
%To remove noisy data, it is good to apply operators that prune away superfluous parts of the data
%and learned model. This study recommends two such operators: the back-select techniques of RIPPER and the 
%WRAPPER feature subset selector. 
%
%
%\subsubsection{Define Research Targets}
%
%Many techniques discussed widely in the SE literature (e.g. formal methods)
%have been rejected by this analysis as not cost-effective. It is important to stress
%that our conclusions are {\em only} for IV\&V. Embedded V\&V is performed in a different
%business context and, in that context, using (e.g.) model checkers to check for temporal logic
%violations in source code might be particularly cost-effective.
%The lesson here is that theoretical SE research needs to understand more about industrial SE practice.
%For example, proponents of a SE method should be aware of the business contexts
%where their method are applicable or impractical. Ideally, the literature should
%routinely contain research targets 
%such as 
%\begin{quote}
%``when generating  formal models from source code costs {\em less than} (say) 
%10\% of the source code generation,
%then formal methods will become practical for (one of) \{agile development, embedded V\&V, etc\}''.
%\end{quote}
%This research target should be justified using industrial data. We would argue that funding bodies
%should identify such research targets, then selectively fund research according to how well
%a research proposal might contribute to those targets.
%
%
%\subsubsection{When to Perform IV\&V}
%

%
%\subsubsection{Collect $\Delta$ Data}
%
%\fig{phase} showed that the relative merits of concept-phase vs requirements-phase IV\&V is crucially dependent on the
%cost-to-fix
%$\Delta$ value. At $\Delta=2$, requirements IV\&V is more valuable than concepts IV\&V (for both
%robotic and manned missions)
%but
%at $\Delta=5$, the pattern reverses and concept IV\&V is more valuable (particularly for manned missions). 
%Organizations should routinely
%collect data like \fig{dabney06a} in order to find their own most cost-effective
% phase to perform IV\&V.
%
%\subsubsection{IV\&V Task Selection}
%
%If budgets are tight, we advise skipping late life cycle activity and focus more on the cheaper
%early life cycle activities. In this context, five tasks are worthy of special attention.
%
%According to \fig{allocated},
%the cheapest and earliest life cycle activities are 
%concept document evaluation (2.4), sw/user requirements allocation analysis (2.5),
%and traceability analysis (2.6).
%Since these tasks are so cheap, and offer the most benefit earliest in the life cycle,
%we would recommend that they are {\em always} 
%performed.
%
%Also, as shown in \fig{allcosts},
%two tasks have an outstandingly
%large cost: interface analysis, code (5.3); and simulation analysis (6.3). Based
%on the above, we would argue that these most expensive, late life cycle, least cost-effective
%tasks should only be performed if there is a very pressing business case to do so.
%
\bibliographystyle{IEEEtran}
\bibliography{refs}
\begin{figure*}[!b]
\footnotesize
\begin{tabular}{|p{.48\linewidth}|p{.48\linewidth}|}\hline
\bi
\item
{\bf am: Artifact Maturity}
\bi
\item[5:] The project's development artifacts are more than 80\% incomplete
for the current stage of development based upon the project's
selected software development life cycle.
\item[3:]
The project's development artifacts are less than 50\% incomplete
for the current stage of development based upon the project's
selected software development life cycle.
\item[1:]
The project has the appropriate artifacts (less than 10\%
incomplete) for the current stage of development based upon the
project's selected software development life cycle.
\ei
\item
{\bf as: Asset Safety}
\bi
\item[5:] 
complete loss of system.
\item[4:]
complete loss of primary subsystem.
\item[3:]
complete loss of other critical asset.
\item[2:]
partial loss of other critical asset.
\item[1:]
short-term partial loss of other critical asset.
\ei

\item{\bf cl: CMM level}

1,2,3,4,5= CMM level 5,4,3,2,1. 
For a fast
guide to assigning CMM levels, see the discussion on
$pmat$ and ``key process areas'' in~\cite{boehm00b}.
\item
{\bf cx: Complexity}
\bi
\item[5:]
Complex scheduling, dynamic priorities, performance
critical embedded system, complex parallelization,
noisy stochastic data.
\item[3:]
Simple nesting, inter-module control
using decision tables, message passing, middle-ware,
simple i/0 processing (status checks and error processing),
minimal structural changes to input files, functions behave differently
in different models of system operation.
\item[1:]
Straight=line code, no nested structures, simple 
composition via procedure calls or some scripts,
functions operate in only one mode, simple
COTS-DB queries and updates.
\ei

\item
{\bf di: Degree of Innovation}
\bi
\item[5:] Basic principle being explored. Proof
of concept exploration
\item[4:]
Component validation in lab or relevant
environment.
\item[3:]
System prototype demonstration in relevant
environment.
\item[2:] System prototype demonstration 
\item[1:] Proven system concept via several successful
prior projects.
\ei

\item
{\bf do: Development Organization}
\bi
\item[5:] Geographically disperse, multiple managers/ developers/ sub-contractors.
\item[4:]
As above, but one sub-contractor.
\item[3:]
As above, but no sub-contractors.
\item[2:]
All personnel co-located, sub-contractors used.
\item[1:]
All personnel co-located, no sub-contractors.
\ei

\item{\bf dt: Use of Defect Tracking}
\bi
\item[5:] The project has no or a minimal defect tracking system.		
\item[3:]
The project has a defect tracking system that is 
not consistent across the project.		
The defect tracking system is implemented in a manual means (e.g. 
a spreadsheet).
\item[1:]
The project has an integrated defect tracking system that 
allows defects to be assigned and corrected along with 
identifying root causes of the defects
\ei

\item{\bf ex: Experience}
\bi
\item[5,4:] less than 5,10 years minimal domain or related experience (respectively).
\item[3:]
10+ nominal domain or related experience.
\item[2:]
developed one like system.
\item[1:]
developed more than one like system or current incumbent.
\ei
\ei
&
\bi
\item
{\bf fr: Use of Formal Reviews}
\bi
\item[5:]
The project has no planned formal reviews.
\item[3:]
The project has planned formal reviews, but no
documented entrance and exit criteria.		
\item[1:]
The project has planned formal reviews throughout the s/w life cycle that include entrance and exit criteria for the review
\ei
\item
{\bf hs: Human Safety}
\bi
\item[5:]
Loss of life; i.e. a component
 receiving this rating is linked to an immediate and direct loss of life.			
\item[3:]
Injury or potential for injury; a component
 receiving this rating has the potential to harm or injure a human.  
It can not directly be linked to a direct loss of life.			 
\item[1:]
Discomfort or nuisance:	a component receiving this rating does not cause injury but may make a human uncomfortable (e.g. some aspects of environmental control) or be a nuisance.			
 \ei
\item
{\bf pf: Performance}
\bi
\item[5:] Unable to achieve minimum objectives/success criteria.
\item[4:]
Unable to achieve many objectives/success criteria.
\item[3:]
Unable to achieve a primary objective/success criteria.
\item[2:]
Unable to achieve a secondary objective/success criteria.
\item[1:]
None of the above
\ei

\item {\bf
ra: Reuse Approach}
\bi
\item[5:] 
Re-use approach is ad hoc and based on similarities between projects with no specific analysis of the item being re-used in terms of applicability to the current project.
\item[3:]
Re-use approach makes use of software artifacts that have been designed for other projects but not with re-use in mind.  These artifacts are analyzed for applicability to the new development effort and the analysis is documented.
\item[1:]
Project has a documented re-use approach that is drawing from an establish re-use repository of software artifacts.
\ei
\item
{\bf ss: Size of System}
Measured in KSLOC. 1,2,3,4 = under 10K,50K, 400K, 850K.
Otherwise, 5.

\item
{\bf uc: Use of Configuration Management}
\bi
\item[5:] Configuration management process is not documented/ not 
integrated into the software engineering process 		
\item[3:]
The project's approach to configuration management is not consistent across the project.  		
Implementation is through manual methods.
\item[1:]
Project has a well-defined and documented process that maintains the integrity of the development artifacts through out the software life cycle through the use of a tried configuration management tool set.
\ei

\item{\bf us: Use of Standards}
\bi
\item[5:] Ad hoc development, minimal documentation and planning
and management.
\item[3:]
Using a documented standard and can demonstrate
partial compliance to that standard. 
\item[1:]
Integrated approach to sw/ development
using a industry wide or locally proven process,
documentation that explains
modifications to that standard (if any).
demonstration of compliance with the
stated standard or modified standard.
\ei
\ei\\\hline
\end{tabular}
\caption{Ranges for SILAP and SILAP2 raw features.}\label{fig:sranges}
\end{figure*}

\input{tim/bio} 
\input{markland/bio} 
\input{ken/bio} 
\input{christina/bio} 
\input{melissa/bio} 
\input{julian/bio} 
\newpage
\appendix
 
The raw SILAP features are scored 5,4,3,2,1  representing
what the SILAP authors~\cite{costello05} believed were
the {\em worst} to {\em best} cases (respectively). \fig{sranges}
offers some notes on how to select the right range for a particular component.


\end{document}

fss ok to throw away much of the data.

the world is cross-correlated.
things that look poor in isolation work well in isolation caution against the single spreadsheet
plot where one feature is plotted against the target variable

se complex, but not some complex that we can't find patterns

se characterizes but much theory and very little experimental verification. 

