\documentclass[twocolumn,global]{svjour}
\usepackage{lastpage,url,alltt,graphicx,cite,url,fancyheadings,bar,fancyvrb}
\usepackage{rotating,wrapfig,stfloats}

\usepackage{eso-pic}
\usepackage{color}
\usepackage{type1cm}

%%\makeatletter
%%  \AddToShipoutPicture{%
%%    \setlength{\@tempdimb}{.5\paperwidth}%
%%    \setlength{\@tempdimc}{.5\paperheight}%
%%    \setlength{\unitlength}{1pt}%
%%      \makebox(600,900){\rotatebox{45}{\textcolor[gray]{0.80}{\fontsize{9cm}{9cm}\selectfont{Draft}}}}}
%%

\newenvironment{ccode}{\scriptsize\begin{alltt}}{\end{alltt}\normalsize}


   \newcommand{\code}[3]{%
     \begin{figure}[!t]%
        \fvset{  samepage=true,%
                 fontsize=\scriptsize,%
                 commandchars=\\\{\},%
                 framesep=2mm,%
                 label={\small{\textit{#1}}},%
                 labelposition=topline,%
                 %numbersep=1pt,%
                 %stepnumber=5%
                 %numbers=left,%
                 frame=single%
                 }%
            {\bf \VerbatimInput{#1}}
    \caption[#2]{#3}\label{fig:#1}%
    \end{figure}%
}

%\usepackage[para,symbol*]{footmisc}
%\setfnsymbol{lamport}
\usepackage{amssymb,times}
\DeclareFontShape{OT1}{cmtt}{bx}{n}{
  <5><6><7><8><9><10><10.95><12><14.4><17.28><20.74><24.88>cmttb9}{}

\newcommand{\mytitle}{
Improving IV\&V Techniques Through the Analysis of Project Anomalies:\\
LINKER - preliminary report
}
\footrulewidth 0.0pt\headrulewidth 0.1pt
\lhead{}\rhead{}\chead{}\rfoot{}\lfoot{}\cfoot{}
\rhead{ {\scriptsize 
\date{January 3, 2007}\newline
source: \url{http://now.unbox.org/all/trunk/doc/07/tellingmore/report3.*}\newline
download: \url{http://menzies.us/pdf/07anomalies-linker0.pdf}
}}

\rhead{{\scriptsize 
\date{January 3, 2007}\\
download: \url{http://menzies.us/pdf/07anomalies-linker0.pdf}\\
wp: \url{http://now.unbox.org/all/trunk/doc/07/tellingmore/report3.*}
}}
\lhead{\includegraphics[width=6cm]{wvulogo.pdf}}

\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{itemize}
   \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{itemize}}

\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}

 \newtheorem{recommend}{Recommendation}

 \newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}
 \newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
 \newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}

%\topmargin -3cm

\begin{document}
\title{
\mytitle
\author{Tim Menzies }
\institute{Lane Department of Computer Science and Electrical Engineering, West Virginia
University,
USA
\newline
\url{tim@menzies.us}\newline
\url{http://menzies.us}\newline
}}
\maketitle \thispagestyle{fancy}
\pagestyle{fancy}

\vspace{-1cm}\begin{abstract}
This project is in two parts.
The {\em second part} will try to combine two (or more) of the IV\&V data sources
 into an active monitoring
framework where data collected during an active IV\&V project will trigger an
alert
if a project becomes unusual'' (and defining ``unusual'' is one of the goals of this project).

Before we can generalize between sources, we need to study each source in isolation
to determine its strengths and weaknesses.
Hence, the {\em first part} of this project  aims to gain experience with the various IV\&V data sources
available to researchers like myself; i.e.
		 \bi
	 \item
	 SILAP, from
	 the IV\&V planning and scoping team; 
	 \item
	 James Dabney's Bayes networks that
	 describe the IV\&V business practices of the L3
 IV\&V contractor; 
	 \item
	 The PITS issue tracking data; 
	 \item
	 The LINKER database
	 project that intends to
	 join PITS to other data sources; 
	 \item
	 Balanced score card strategy maps from NASA Langley. 
	 \item
	 and the COCOMO data sets from JPL.
	 \ei

\noindent
To date, this project has resulted in:
\bi
\item
A preliminary report described what had been learned from the
SILAP data. 
A ranking was offered on the most common IV\&V work-breakdown structure (WBS) activities.
This ranking can be used for (e.g.) identifying what WBS tasks would benefit most from optimization.
\item
A preliminary report
Dabney's Bayes networks. On a limited case study, it was shown that Bayes nets
and treatment learning could generate parsimonious explanations 
for project events.
\item
A preliminary report on text mining from the
PITS issues tracking database that 
generated an expert system
which audited  a test engineer's proposed severity level.
\ei
This report describes LINKER, a new database at IV\&V that combines data from different sources.
LINKER is still under-development. Hence, unlike the prior reports, this report 
does not report experimental results. Rather, it discusses the benefits and drawbacks of LINKER.
\end{abstract}
\newpage
\noindent
{\bf Credits:}
This report was generated with the assistance of Wesley Deadrick (computer engineer at NASA IV\&V).
This research was conducted at
West Virginia University 
under NASA sub-contract project 100005549, task 5e, award 1002193r.

~\\

\noindent
{\bf Cautions}
Reference herein to
any specific commercial product, process, or service by trade name,
trademark, manufacturer, or otherwise, does not constitute or imply
its endorsement by the United States 
Government.

~\hrule~\\
\begin{figure*}[!b]
\begin{center}
\includegraphics[width=5in]{linker/pits.png}
\end{center}
\caption[PITS]{The login screen to the PITS database.}\label{fig:pits}
\end{figure*} 

\newpage
\rhead{\thepage\ of \pageref{LastPage}}
\lhead{Menzies: LINKER - preliminary report}
\tableofcontents
\listoffigures
\newpage

\section{Introduction}
We seek methods to monitor NASA projects using the database housed at IV\&V.
One such data base is PITS, the
Project and Issue Tracking System shown in \fig{pits}.
\begin{figure*}[!t]
\begin{center}
\includegraphics[width=5in]{linker/importing.jpg}
\end{center}
\caption[LINKER]{LINKER, accessing data from other IV\&V databases.}\label{fig:import}
\end{figure*} 

  
While PITS
 works well for queries over one project, it is a poor interface
for running queries over multiple projects.
Also,
there is a lack of consistency in how each
of the projects collects PITS issue data.
While there was a set of required data fields, the
majorities of those fields do not provide information in regards
to the quality of the issue and are not very suitable for comparing
projects.

NASA is very aware of the problems with PITS and is taking
active steps to improve it.
 At the time of this
writing, there is an on-going effort to implement a mandatory data
set in each IV\&V project database to support IV\&V effectiveness
metrics.  

As part of the effective metrics work, Wesley Deadrick is leading
the LINKER project that aims to act as a centralized query system
for IV\&V data.  LINKER will have read-only access to the other
IV\&V databases and offer a centralized query facility that can
reach into the other databases.  Internally, LINKER works by first
importing data from other IV\&V databases (see \fig{import}).
Initially, LINKER is only targeting PITS but, when other databases
become available, Deadrick's team would integrate them as well.
Once imported, LINKER can then explore data from multiple projects.

This report is an audit of LINKER. Specially, 
we will ask ``how good is LINKER?'':
\bi
\item
As a standard research tool. 
Several drawbacks with the current version of LINKER will  be discussed.
\item
For  detecting anomalies in active
IV\&V projects. 
Our analysis shows that LINKER stores only a minority
of the
data points needed for this anomaly detector.
\ei

However, the point of this report is that LINKER is not some
static, already written, set-in-stone piece of software.  Rather,
it is the focus of an active and motivated development team. 
The
goal of this report is to offer insight to that team into how best
modify the current version of LINKER in order to enable its use in
the anomaly detection process.
To that end, this report:
\bi
\item
Makes 14 specific recommendations about extending the current
version of LINKER.
\item
Discusses what could be done if {\em none} of those recommendations
are implemented- see \tion{solo} about  {\em solo mode}.
\ei
\section{LINKER as a Research Tool}

For the purposes of this report, ``researchers'' means {\em either}
NASA civil servants looking for IV\&V-specific issues {\em or}
university researchers seeking general methods for improving software
engineering.  This section discusses the merits of LINKER for use
by researchers.

LINKER was designed to overcome certain limitations with PITS. The
current interface of PITS can only run queries against a single
project. There is no way  to select multiple projects or to save a
query generated on one screen for one project and to use it on
another project.

Once projects are selected, LINKER offers a set of canned queries
for report generation.  While such canned queries are a useful way
to introduce novices to  a system, they cannot capture the space
of possible research queries:
\bi
\item
 The merits of canned-vs-arbitrary
queries was demonstrated recently at a review for a WVU 
university
initiative (December 13, 2006). 
\item
Two WVU researchers discussed
their work on a two year cycle of ``get data from project'' and
``consider what to ask for next''.
\item
 They found that every data dump
from the projects offered hints on what other information might be
most informative.
\item
In fact, to complete their research, they needed to do large scales dumps of the data
followed by intricate, special purpose queries.
\ei
The lesson of the above is clear:
\begin{quote}\input{r2}\end{quote}
\begin{quote}\input{r12}\end{quote}
\begin{quote}\input{r1}\end{quote}

(Regarding 
{\bf Recommendation 2}: Deadrick comments that
the LINKER team is currently designing a ``configurable
query tool'' which allows users to construct queries on the
fly which span multiple projects. Due to
inconsistencies in the PITS database,
this is proving to be a challenging 
task.)

Quite apart from the technical issues, PITS data must be kept secure.
Access to project data can be controlled on a project-by-project basis and accessing
all data on all projects might involve an elaborate and extended administrative
task of collecting the required permissions.

\begin{quote}\input{r13}\end{quote}

One solution to this security issue would be to filter all data from
PITS via a set of {\em synonym tables} that
obfuscate the specific project data, perhaps at increasing levels of security.

\begin{quote}\input{r3}\end{quote}

(Regarding {\bf Recommendation 5}: Deadrick comments
that a synonym table for all LINKER fields might be
overly-resource expensive. However, if synonyms are
restricted to just the fields
for  ``issue counts'', ``severities'', ``subsystem'', 
``WBS activities'' and ``tasks'', ``phase found'', 
and ``phase introduced'', then these synonyms
could become quite simple.)

\begin{quote}\input{r4}\end{quote}
\begin{quote}\input{r5}\end{quote}
\begin{quote}\input{r6}\end{quote}
\begin{quote}\input{r7}\end{quote}
\begin{quote}\input{r8}\end{quote}

Once the synonyms are in place, then LINKER can be made broadly available:

\begin{quote}\input{r14}\end{quote}

\section{LINKER for Anomaly Detection}

Having assessed the current version of LINKER, this report now discusses how
it might be extended to support anomaly detection.
Anomaly detection is at least a three-stage  process:
\bi
\item
 In the first stage,
some {\em expectation} is developed for observable values. 
A requirement for this first stage to work is some {\em relevant background knowledge} of the domain.
Our current best candidate for {\em background knowledge} are the Bayes
networks developed by James Dabney et.al.~\cite{dabney06}
describing the  practices of an IV\&V contractor. One advantage of Bayes nets
are that they can handle uncertain information. Also, their internal distributions can be incrementally refined
based on updated domain information. That is, these nets can be kept {\em relevant}  to changing conditions, using
an automatic process.
\item
In the second stage,
some {\em observables} are collected from the domain.
A requirement for this second stage is access to an  {\em active data source}.
\item
In the third state, the probability of the observables is computed
using the expectations. If that probability is low, then an anomaly
is detected.  Of these stages, this  stage is the simplest to
achieve. Decades of AI research has resulted in Bayes classifier
technology~\cite{domingos97optimality,john95}, that  compute the
expected likelihood of prior projects, and detects when a new example
is an outlier from the past~\cite{me06c}.  
\ei

Since we have some confidence in stages one and three, this report
will focus on stage two. We ask the questions ``is LINKER a candidate
{\em active data source}?'' 
We will seek holes in LINKER's schema by ``asking'' it what it can say about
the background knowledge represented in Dabney's Bayes nets.
Therefore, to assess LINKER we first need to understand Dabney's belief nets.

\subsection{About Belief Networks}

A belief network (also known as a Bayesian network or a Bayes net)
is a directed acyclic graph of 
\bi
\item
nodes representing variables (nodes can represent any kind of 
variable, be it a measured parameter, a latent variable or an hypothesis);
\item
arcs representing probabilistic dependency relations among the variables 
and local probability distributions for each variable given values of its parents.
\ei

If there is an arc from node $A$ to another node  $B$, then variable $B$ 
depends directly on variable $A$, and $A$ is called a $parent$ of $B$. 
If there are $n$ variables, then
for each variable $X_i$ ($1 \le i \le n$),
the set of parent variables is denoted by $parents(X_i)$ 
then the joint distribution of the variables is product of the local distributions
\[
Pr(X_1, \ldots, X_n) = \prod_{i=1}^n Pr(X_i \mid parents(X_i))
\]
If $X_i$ has no parents, its local probability distribution is said
to be $unconditional$, otherwise it is $conditional$. If the variable
represented by a node is $observed$, then the node is said to be
an $evidence$ node.

\begin{figure*}
\begin{center}
\includegraphics[width=7in]{linker/bayesall.png}
\end{center}
\caption[A Bayes net of IV\&V practices]{Dabney's 
Bayes net describing IV\&V practices.
The red square, top left, is expanded in \fig{bayes1}.
}\label{fig:bayes}
\end{figure*} 
\begin{figure*}
\begin{center}
\includegraphics[width=5in]{linker/bayes1.png}
\end{center}
\caption[Defect introduction]{The ``defect 
introduction'' Bayes sub-net shown in a red box, top-left of \fig{bayes}.}
\label{fig:bayes1}\end{figure*} 

\subsection{Belief Networks for IV\&V}

Using this representation, Dabney described the 
IV\&V practices of a NASA contractor. That description comprised:
\bi
\item
The Bayes net of \fig{bayes};
\item Distributions 
for the inputs to the network for three different NASA projects.
\ei

\noindent
\fig{bayes} may appear  complex, but its structure is actually
quite simple:
\bi
\item
  The
{\em rows} represent the actions of developers and IV\&V analysts working
on NASA software. Row one shows developers building software and,
inadvertently, introducing defects.  Developers are well aware of
their shortcomings so, in row two, they perform numerous defect
removal activities.  In order to ensure software quality, NASA runs
an IV\&V team which (in row three) also removes defects.
\item
The {\em columns} represent a standard NASA software development project:
\begin{footnotesize}
\[ requirements \rightarrow design \rightarrow code \rightarrow test \rightarrow integration
\]
\end{footnotesize}
\item
The {\em horizontal thick arrows} in the diagram model defects generated
at each phase leaking into subsequent phases. 
\item
The {\em vertical thick arrows} model defects being generated, then removed by
either the developers or the IV\&V team.
\ei
In terms of understanding Dabney's networks, the essential
feature of \fig{bayes} is that each column stores essential
the same structure, with minor differences; e.g. 
occasionally a node might have three parents,
not two. 
Hence, reasoning about much of the system can be simply achieved via
a closer examination of any of the columns.
\begin{figure*}
\begin{center}
\includegraphics[width=4.5in]{linker/bayes2.png}
\end{center}
\caption[Defect removal (developer)]{The developer ``defect removal'' 
Bayes sub-net (middle row of \fig{bayes}).}\label{fig:bayesdrd}
\end{figure*} 

\begin{figure*}
\begin{center}
\includegraphics[width=4.5in]{linker/bayes3.png}
\end{center}
\caption[Defect removal (IV\&V)]{The IV\&V ``defect removal'' 
Bayes sub-net (bottom row of \fig{bayes}).}\label{fig:bayesdri}
\end{figure*} 

\subsection{Assessing LINKER}

Having described all that, the next question is ``how many nodes in
\fig{bayes1} and
\fig{bayesdrd} and
\fig{bayesdri}
can we observe in actual projects?''. 
In this regard, 
the `Techniques Employed''
subset (top right of \fig{bayesdrd}) is particularly insightful.
If details of  tool use were routinely stored in the
IV\&V project monitoring databases, then those fields might never be filled in.
For contractual reasons, the IV\&V database does not include information about the
specific tools used by IV\&V contractors. The general rule is that NASA civil servants
describe ``what'' is to be done and the contractors are free to decide ``how'' to do it,
using their available resources. 

Similarly, the contractors might object to offering data on the ``Staff Ability'' node (middle, left hand side, \fig{bayesdrd}) since they
might require offering too much information to  the civil servants. 

In fact, a detailed of review of
\fig{bayes1} and
\fig{bayesdrd} and
\fig{bayesdri} by Menzies and Deadrick showed that only a minority of the nodes were directly 
observable (either using LINKER or any other known candidate source).
In the defect introduction subnet of \fig{bayes1}:
\bi
\item {\em System Document Quality} might be accessible via the new Cukic WVU University initiative;
\item {\em Requirements Stability} might be accessible via a configuration 
management tool that recorded the rate of change in the requirements documents;
\item {\em Novelty of Problem Approach} could  be read from the SILAP planning and scoping database
or a COCOMO description of the project;
\item {\em Staff Experience Level} might be accessible if contractors could release the years of
service of the contractors assigned to this IV\&V task;
\item {\em Schedule Pressure} could be read from a COCOMO description of the project;
\ei

\noindent
In the developer defect removal subset of \fig{bayesdrd}:
\bi
\item {\em Review Quality} might be accessible via LINKER and the number of issues found;
\ei

\noindent
In the IV\&V  defect removal subset of \fig{bayesdri}:
\bi
\item {\em Data Availability} might be accessible from knowledge of the documents currently in-house at IV\&V
(for example, using ``phase alignment'').
\item
{\em Program Acceptance} can be
accessed in PITS/LINKER by recording how often
issues were accepted, disputed, withdrawn,etc.
\item {\em Experience Level} might be accessible if contractors could release the years of
service of the contractors assigned to this IV\&V task;
\item {\em Staffing Level} might be accessible from the Planning \& Scoping databases;
\item {\em Schedule Pressure} could be read from a COCOMO description of the project;
\ei

Recall from the above discussion that 
\fig{bayes} contains five columns, each of which holds the 63 nodes (plus or minus
a few) shown in 
\fig{bayes1},
\fig{bayesdrd} and
\fig{bayesdri}. Assuming the above pattern repeats across the five columns,  then
Menzies and Deadrick 
review of all the data sources from all the current IV\&V  projects shows that we might be
able to collect for 5*11=55 of the 5*63=315 nodes in \fig{bayes} ($\frac{55}{315}\approx17\%$)
of the nodes.

The good news is that 17\% might be enough. Bayes nets are well-defined
mathematical entities 
that come with auto-update tools. Given background expectations of the
these nodes, small drips of data (say, 17\%) can be enough to calibrate
the nets and let them make sensible predictions about expected and
anomalous behavior.

The bad news is that the current version of LINKER  can only supply data
on
$5*\frac{2}{11}$\footnote{{\em Review Quality} and {\em Program Acceptance}}
of the nodes. To obtain the remaining data,
 LINKER's current functionality would
have to be extended to collect data from more inputs.

\begin{quote}\input{r9}\end{quote}
\begin{quote}\input{r10}\end{quote}

\noindent
More generally, there are too many ``island databases'' at IV\&V and they need to be
linked together:

\begin{quote}\input{r11}\end{quote}

\section{Solo Mode}\label{sec:solo}

Every programmatic change consumes resources. If the recommendations listed above cannot be make,
what is the fate of this anomaly detection project? Can it proceed?

The answer is ``yes'', but in a modified form. Currently, the concept
of operations is that the Bayes nets are wired into the projects
via LINKER. In that mode, the nets are automatically triggered by
any update to the project information.  We call  this the ``integrated
mode''.

In an alternate ``solo mode'', the anomaly detector is a stand-alone
application and the user is responsible for manually entering in
the node data. In this ``solo mode'', the NASA project manager is the API that connects
the anomaly detector to the projects. 

While the solo mode is not the preferred option, is it possible and practical:
\bi
\item
 Recall from the above
that Bayes nets can work from small amounts of imprecise data. If once a month, the manager spent 20 minutes
with the tool entering in the available data, then they could sufficient for anomalies to be detected. 
\item
Under the hood, the data entered by the manager could be stored away in a database, so results from different projects could be compared and combined. 
\item
 Data miners could run over the models looking for the {\em least} number of nodes that the user has to enter in order to describe the current tactical situation (this would reduce the data entry effort of the user).
\ei

In fact, solo mode could be used to bootstrap integrated mode. If the recommendations made here are
too expensive to be performed in the short term, then a successful track record with solo mode could be used
to lever funds to make the required changes to LINKER which, in turn, would enable integrated mode. 
\newpage
\bibliographystyle{abbrv}
\bibliography{refs}
\end{document}


