\PARstart{T}{he} analysis of this paper is very different
to the empirical software methods proposed
by prior publications that used NASA data.
It is informative to the industrial realities of IV\&V
to understand the reasons for 
those differents.
 
At NASA Software Experiemence Laborotory (SEL)~\cite{basili02,basili88,basili99},
an initial {\em data collection stage} lasted six years from 1976 to 1983. Experiments
in this phase were of three types:
\bi
\item
{\em screening experiments}
(understand what's going on now) 
that
baselined current  projects, 
\item
{\em semi-controlled experiments}  comparing different 
methodologies on different related projects, 
\item
{\em controlled experiments}
conducted at the SEL, replicating the same project using 
different methods.
\ei
Three of the major lessons~\cite{basili02} of that period were:
\bi
\item
{\em Lesson 1:} rigorous data collection adds 10\% to project cost;
\item {\em Lesson 2:} you must compromise in asking for only as much information as is feasible to obtain;
\item
{\em Lesson 2:} subjective data was important in understanding the impact 
of a methodology. 
\ei

After a six year data collection phase, the SEL changed
 into a {\em learning organization}
that studied how to measure the impact of applying specific technologues (e.g. ADA, IV\&V).
At the SEL, this new phased was characterized by:
\bi
\item An {\em experience factory} (EP) that systematically saved and reused experience from
previous projects;
\item A {\em quality imporvement paradigm} (QIP) that  applied the scientific method
to software engineering;
\item The {\em goal, question, metrics} (GQM) paradigm  where collected data was tuned to
a set of objects needed to address a stated goal.
\ei

The SEL's learning organization phase lasted ten years (from 1984 to 1994)
during a
period of relative stability within the NASA organization. 
Starting in 1995, the pace of change within NASA increased.
New projects were often out-sourced and the SEL became less the
driver and more the observer, less proactive and more reactive
to software development actitivies. Without an owner of
the software development process, each project could adopt its
own structure. SEL-style experimentation became difficult sinse
there was no longer a central model to build on.
Basili et.al.~\cite{basili02} describes this 1995-2001 phase as one
of ``retrenchment'' at the SEL\footnote{At the time of this writing,
the SEL still exists but with a greatly reduced role within NASA
and without the staff seen in the 1976-2001 period.}.

The pace of change that was so fatal to the SEL's experience factory
has not decreased:
\bi
\item
In the 1990s, NASA moved to  a ``faster, better, cheaper'' paradigm that
lead to massive changes in NASA's development practices. For example, prior to 1990, NASA's Jet Propulsion
Laboratory launched (on average) five spacecraft a year, all of which were built internally. In the 1990s,
JPL launched dozens of craft, most of which were built by external contractors. 
\item
This policy lead
to certain high profile errors which was attributed to production haste, poor communications, and
mistakes in engineering management~\cite{davis00}. Accordingly, NASA abondonned  Faster, Better Cheaper 
and changed development methods. 
\item
In 2003, the loss of the Challenger shuttle during re-entry prompted
another major agency reorganziation. Development budgets were significantly altered to fund the ``return to flight''
program that lead to the successful re-launch, in 2005, of the shuttle program.
\item
As the time of  this writing, NASA is significantly reorganizing  (again) around the ``mission to Mars''.
\ei
NASA's numerous national reorganizations  have been reflected in the
IV\&V program.
\fig{hist} shows that
three times (in 1991, 1994, 2000),
the oversight for IV\&V has changed
from NASA headquarters then to two centers on different
sides of the country.
%It is insightful to consider
%the resasons for these changes.
%As befits an organization that launches humans into space,
%NASA's management is appropriately
%cautious about imposing major management changes.
%Initially,
%NASA implemented IV\&V only on a few projects (Space Shuttle,
%International Space Station). 
Also,
in 1996, 
NASA develop new contractual methods that allowed IV\&V to be
applied to a wide range of other projects.
These new methods resulted in  large change to IV\&V management practices
as the focus expanded from a few on-going projects to dozens of smaller
projects.

In 1999, after 11 years 
of success,
IV\&V was mandaded
for the entire NASA organzation. 
But a problem remained-
projects had to find funds for IV\&V from
within their own budgets. This  often slowed IV\&V
work since
managers were forced to juggle the demands of
developments with the demands of IV\&V.
It took four more years of successful IV\&V before NASA
could solve this problem.
In 2003, NASA reaffirmed and stregthened its support for IV\&V
allowing
projects
to fund their IV\&V from a headquarters'  source.

The above changes meant that more projects could receive the benefits of IV\&V.
However, the same changes that benefitted NASA also
complicated
the twin goals of
data collection and analysis.
In nearly all projects,
the schema of IV\&V's issue tracking
sytems was 
modified to better address some specific requirement.
While some data fields were required,
most did
were not suitable for comparing
the cost-effectiveness of IV\&V across projects.

In summary,  NASA is similar to many
other large organizations that constant adapt their practices
and their databases as their
business changes. 
Were possible, database schemas are standardized across multiple proejcts and multiple years.
However,  NASA's record keeping is in as much flux as the rest of
the organization and no data is collected consistently across all projects and all years.

Due to the pace of organization change,
this study could not use
the experience factory approach pioneered at the
NASA SEL.
The stability required for such an analysis is missing.
Hence 
we cannot reasonably expect to
conduct the {\em semi-controlled} or {\em controlled} experiments
described above. 
Also,
recalling Lesson 1 (above), rigorous
SEL-style data collection adds another 10\% to the project cost.
IV\&V is typically costed at 10\% (or less)
of the software development costs~\cite{lewis90}. 
That is, 
SEL-style rigourous data collection
it would {\em double} the cost of IV\&V. 

Accordingly, following Lesson 2 (only use the information that
is feasible to obtain), the rest of this paper focuses on what can be learned
from the 
data currently available in the IV\&V databases.  
NASA is well aware of the impact of organization change on its databases.
Hence, there is an on-going current effort to implement a mandatory data
set in each IV\&V project.
For example:
\bi
\item
NASA now uses a standard five-point scale  to score issue severity.  A
slightly different scale is used for robotic and human-rated missions
(see \fig{robots} and \fig{humans}) but the interpretation of the
scales is same across all missions: severity one issues imply total
disaster while severity five issues are ignorable.
\item
From 2005, it was required
that a standard set of project factors
are collected at the start of every IV\&V project. These
16 {\em SILAP factors}~\cite{costello05} are discussed below.
\ei
\input{tim/severity}
Accordingly, the research challenge of this report was to make the most of the available data;
i.e. the SILAP data collected since 2005 mapped to the issue reports.
Data mining will be used
to find a predictor 
for {\em when} a project has a high (or low)
likelihood of generating high severity issues.
Simple visualization methods will then we used to learn {\em what} tasks to apply.


