\PARstart{D}{ata} analysis of a national software program 
must be sensitive to certain features of large scale data collection.
To understand those features, it is best to contrast them with standard empirical methods.

One set of standard methods
were those  developed at the NASA Software Experiemence Laborotory (SEL) in the period
1976 to 1994~\cite{basili02,basili88,basili99}.
At the SEL,
an initial {\em data collection stage} lasted six years from 1976 to 1983. Experiments
in this phase were of three types
\bi
\item
{\em screening experiments}
(understand what's going on now) 
that
baselined current GSFC projects, 
\item
{\em semi-controlled experiments}  omparing different 
methodologies on different related projects, 
\item
{\em controlled experiments}
conducted at the SEL, replicating the same project using 
different methods.
\ei
Three of the major lessons~\cite{basili02} of that period were:
\bi
\item
{\em Lesson 1:} you must 
compromise in asking for only as much information as is 
feasible to obtain;
\item
{\em Lesson 2:} rigorous data collection adds 10\% to project cost;
\item
{\em Lesson 3:} subjective data was important in understanding the impact 
of a methodology. 
\ei
 
After a six year data collection phase, the SEL reorganized into a {\em learning organization}
that studied how to measure the impact of applying specific technologues (e.g. ADA, IV\&V).
At the SEL, this new phased was characterized by:
\bi
\item An {\em experience factory} (EP) that systematically saved and reused experience from
previous projects;
\item A {\em quality imporvement paradigm} (QIP) that  applied the scientific method
to software engineering;
\item The {\em goal, question, metrics} (GQM) paradigm  where collected data was tuned to
a set of objects needed to address a stated goal.
\ei

The SEL's learning organization phase lasted ten years during a
period of relative stability within the NASA organization. 
Starting in 1995, the pace of change within NASA increased.
New projects were often out-sourced and the SEL became less the
driver and more the observer, less proactive and more reactive
to software development actitivies. Without an owner of
the software development process, each project could adopt its
own structure. SEL-style experimentation became difficult sinse
there was no longer a central model to build on.
Basili et.al.~\cite{basili02} describes this 1995-2001 phase as one
of ``retrenchment'' at the SEL. At the time of this writing,
the SEL still exists but with a greatly reduced role within NASA
and without the staff seen in the 1976-2001 period.

\fig{hist} shows that
the pace of change within NASA has not decreased.
Three times since the start of NASA's IV\&V work,
the oversight for IV\&V has changed
from NASA headquarters then to two different centers on the west
coast and east coast of America. It is insightful to consider
the resasons for these changes.
As befits an organization that launches humans into space,
NASA's management is appropriately
cautious about imposing major management changes.
Initially,
NASA implemented IV\&V only on a few projects (Space Shuttle,
International Space Station). IV\&V was mandaded
for the entire NASA organzation only after 11 years of
experience showed the value of using external reviewers
to assess code.  
But a problem remained-
projects had to find funds for IV\&V from
within their own budgets. This  sometimes blocked IV\&V
work since
managers were forced to juggle the demands of
developments with the demands for external review.
It took four more years of successful IV\&V before NASA
could solve this problem.
In 2003, NASA reaffirmed and stregthened its support for IV\&V
allowing
projects
to fund their IV\&V from a central agency source.

There were four effects
of this constant change The net result of all this institutional change was a continual
change in the data collected at IV\&V. 

 The increased number of projects receiving IV\&V

was only made possible
by NASA making the most of its available resources.
Organizing a national IV\&V program is a resource intensive
activity. 


Only after the success of
that work 

program started hence started out  
cautious in Washireacting to 
As conditions change, NASA has adapted by reorganizing
its resources in order to increase  has 

Many of the day-to-day activities of the SEL involved the data 
capture activities of the developers. Getting their support was an 
important early step. At first there was considerable reluctance 
to cooperate: 

\bi
\item Screening experiments (to understand what is going on) to create baseline measures;
\item 
and included: screening experimnets
used here is 
