%I really liked the numbering of lines! It makes it much easier to state
%errors and relate them to the paper. 11: 'automatically' -> 'automatic' 25:
%'Bayes' -> 'Bayesian' 63: 'don't' -> 'do not' 93: What the hell is
%'summarization'? 64: delete 'the' 135: 'to' -> 'too' footnote 3: space
%before 'Z', full stop at end 224: full stop before '(' 226: invert '.' and
%')' 228: delete extra 'the' 236: What the hell is 'deprecated'? 239:
%'implementin' -> 'implement' 240: delete extra 'of' 241: 'onto' is not
%right 275: What the hell is 'persevered'? 285: delete first 'to' 396:
%insert 'are' after 'there' 400: 'out' -> 'from' 403: insert 'the' after
%'compute' 410: 'to' -> 'the' 413: 'N-descrete' -> 'N discrete' 419: delete
%'by' 461: 'it's' -> 'its' 496: 'subtely' -> 'subtly' 560: 'for' -> 'from"
%624: 'scare' -> 'scarce'


\documentclass[10pt,twocolumn]{article}
\usepackage{../../tex/latex8,../../tex/syntax,../../tex/cite}
\usepackage{times,alltt,url,graphicx,amssymb,../../tex/bar}
\usepackage{../../tex/lineno}%\runninglinenumbers\modulolinenumbers[10]

\clubpenalty=9999 \widowpenalty=9999
%\footrulewidth 0.0pt\headrulewidth 0.0pt
%\lhead{}\rhead{}\chead{}\rfoot{}\lfoot{}\cfoot{}
%
%\rhead{\includegraphics[height=0.5in]{ASEbiglogo.eps}}
%\lhead{{\footnotesize { Submitted to the 16th IEEE International
%Conference on Automated Software Engineering
%                                                                          Nov
%20-22, San Diego, USA.\newline
%\url{http://sigart.acm.org/Conferences/ase/}\newline WP ref:
%01/choices/ \today }}} \headrulewidth 0.4pt

%\documentstyle[alltt,icse2000,twocolumn,url,bar,epsfig,times,alltt,array,amssymb,graphicx,ifthen,color]{article}

% Changes:
%  Page limit: 10 pages
%  Abstract limit: 200 words
%  Contact information is now gall@infosys.tuwien.ac.at
%  URL for author kit changed: needs to be filled in.
%  All other font and sizing information is the same:
%    check whether it still applies.

%\makeatletter
%\def\fnum@figure{\sffamily\figurename~\thefigure}
%\def\fnum@table{\sffamily\tablename~\thetable}
%\makeatother

\newenvironment{smallitem}{
    \setlength{\topsep}{0pt}
    \setlength{\partopsep}{0pt}
    \setlength{\parskip}{0pt}
    \begin{itemize}
    \setlength{\leftmargin}{.2in}
    \setlength{\parsep}{0pt}
    \setlength{\parskip}{0pt}
    \setlength{\itemsep}{0pt}}{\end{itemize}}

\newenvironment{code}{\begin{alltt}\small}{\normalsize\end{alltt}}

\newenvironment{ccode}{\begin{code}\begin{center}}{\end{center}\end{code}}




\begin{document}

\title{Better reasoning about software engineering activities}

\author{Tim Menzies$^1$\\University of British Comlumbia\\{
\url{tim@menzies.com}} \and James D. Kiper$^2$\\Miami
University\\{\url{kiperjd@muohio.edu}}}




\newcommand{\DO}  {{\sffamily \{}}
\newcommand{\FOR}{{\underline {\sffamily for}}}
\newcommand{\DONE}{ {\sffamily \}}}
\newcommand{\IF}{{\underline {\sffamily if}}}
\newcommand{\FI}{{\underline {\sffamily fi}}}
\newcommand{\THEN}{{\underline {\sffamily then}}}
\newcommand{\ELSE}{{\underline {\sffamily else}}}
\newcommand{\WHEN}{{\underline {\sffamily when}}}
\newcommand{\UNTIL}{{\underline {\sffamily until}}}
\newcommand{\AND}{{\underline {\sffamily and}}}
\newcommand{\TO}{{\underline {\sffamily to}}}
\newcommand{\IS}{ {\sffamily $\leftarrow$}}

\newcommand{\NIL}{ {\sffamily $\emptyset$}}
\newcommand{\REPEAT}{{\underline {\sffamily repeat}}}
\newcommand{\RETURN}{{\underline {\sffamily return}}}
\newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
\newcommand{\be}{\begin{enumerate}}
\newcommand{\ee}{\end{enumerate}}
\newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}

\shortverb{!}

\maketitle
%\copyrightspace
%
% If you want to print drafts of the paper with a draft
% notice in the copyright space, comment out the \copyrightspace
% line above and include the \submitspace line below instead.
%
%\submitspace{Draft of paper submitted to ASE 2000.
%\\WP:b/00/ase/whatif(\today).}


% Page number on the initial page can be omitted in both the review and
% final submission (and should be removed in the final submission).  The
% line below does that.

\thispagestyle{empty}  % suppresses page number on first page

% In the review submission, page numbers should appear (they can be omitted
%  from the first page).  The pagestyle command below puts them in.
% In the final submission of accepted papers, page numbers should be
%  omitted; remove or comment out the pagestyle line below to omit them.

\pagestyle{empty}

% Use \section* instead of \section to suppress numbering for
% the abstract, acknowledgements, and references.

\section*{ABSTRACT} Software management oracles often contain
numerous subjective features. At each subjective point, a range of
behaviors is possible. {\em Stochastic simulation}  samples a
subset of the possible behaviors. After many such stochastic
simulations, the TAR2 {\em treatment learner} can find control actions that have
(usually) the same impact despite the subjectivity of the oracle.

%program too simple? XXX space too simple? no- decision tree

\Section{Introduction}

\footnotetext[1]{Proceedings of ASE 2001, San Diego, USA. Corresponding author. Department of Electrical \&
Computer Engineering; 2356 Main Mall; Vancouver, B.C. Canada
V6T1Z4. Phone: (604) 822-3381 Web: \url{http://tim.menzies.com}.}
\footnotetext[2]{Dept. Computer Science \& Systems Analysis, Miami
University, Oxford, Ohio, USA.}

Experience with software management can be encoded into an
management   {\em oracle} that can offer advise on how to
structure a software project. Three components are required for
such an oracle. Firstly, a {\em knowledge source} is required.
Secondly, a {\em syntax} is required to encode that knowledge
source. Thirdly, an interpreter is required to execute the syntax.
In the case of subjective software engineering knowledge, this
interpreter must be able to handle {\em degrees of belief}.

This paper reports experience with one such oracle. The {\em
knowledge source} was  a detailed description of
CMM2~\cite[p125-191]{paulk95}. We used CMM2 since, in our
experience, many organizations can achieve at least this level.
CMM2 is less concerned with issues of (e.g.) which design pattern
to apply, than with what overall project structure should be
implemented. Improving CMM2-style decisions is important since in
early software lifecycle, many CMM2-style decisions effect the
resource allocation for the rest of the project.

The  JANE language was used to {\em encode} the
knowledge source.  In order to handle {\em degrees of belief},
JANE assigns a !Chances! weight to all of its propositions. This
weight is a subjective judgement and so, for each run, the JANE
interpreter hence varies this weight according to distributions
supplied by the analyst. After many such runs, a large log of
possible behaviors is generated which is summarized by the TAR2
{\em treatment learner}.


\begin{figure*}
\begin{minipage}{.75\linewidth}
{\small
\begin{center}
\begin{tabular}{cccc}
\begin{barenv}
\setstyle{\sffamily} \setwidth{8}
  \setyaxis{0}{75}{10}
\sethspace{1} \setnumberpos{down} \bar{14}{1} \bar{22}{2}
\bar{30}{4} \bar{24}{8}
\end{barenv}&
\begin{barenv}
\setstyle{\sffamily} \setwidth{8}
  \setyaxis{0}{75}{10}
\sethspace{1} \setnumberpos{down} \bar{5}{1} \bar{16}{2}
\bar{5}{4} \bar{74}{8}
\end{barenv}&
\begin{barenv}
\setstyle{\sffamily} \setwidth{8}
  \setyaxis{0}{75}{10}
\sethspace{1} \setnumberpos{down} \bar{7}{1} \bar{13}{2}
\bar{19}{4} \bar{60}{8}
\end{barenv}
&
\begin{barenv}
\setstyle{\sffamily} \setwidth{8}
  \setyaxis{0}{75}{10}
\sethspace{1} \setnumberpos{down} \bar{5}{1} \bar{14}{2}
\bar{26}{4} \bar{55}{8}
\end{barenv}
\\~\\
$\Delta_0$=current  &  $\Delta_1$  &
$\Delta_2$ & $\Delta_3$ \\
worth=1 &  worth=1.44 &
  worth=1.31 &  worth=1.28
\end{tabular}
\end{center}
}\end{minipage}\hfill
\begin{minipage}{.25\linewidth}
\begin{raggedleft}
{\scriptsize

\begin{tabular}{|cp{1in}|}\hline
\multicolumn{2}{|p{1.5in}|}{~\newline\underline{{\bf KEY:}}
\newline Top-to-bottom = least

desirable to most desirable.\newline~\newline}\\
\protect\legend1{~}&\begin{raggedright}= high cost, low chances;
i.e. a very bad software \newline project\end{raggedright}
\\
\protect\legend2{~}&\begin{raggedright}= low cost, low
chances\end{raggedright}\\
\protect\legend4{~}&\begin{raggedright}= high cost, high
chances\end{raggedright}\\
\protect\legend8{~}&\begin{raggedright}= low cost, high chances;
i.e. a good software\newline project\end{raggedright}\\\hline
\end{tabular}}
\end{raggedleft}
\end{minipage}
\caption{ Ratios of different software project types seen in four
situations. }\label{fig:output}
\end{figure*}

Despite the range of possible actions within our encoding of
CMM-2, and the subjectivity of the weights measures, there exists
a small number of actions that  have a significant impact on the
project. \fig{output} shows three sets of actions  learnt by TAR2.
The left-hand-side histogram marked $\Delta_0$ shows the ratio of
different project types predicted by our software management
oracle. The other histograms ($\Delta_{i>0}$) show how those
ratios change after applying the {\em treatments} learnt by TAR2;
i.e. managers taking certain actions to change their current
situation. The {\em worth} of each option is a reflection of the
proportion of good and bad projects, compared to $\Delta_0$; i.e.
($worth(\Delta_0)=1$). Note that as {\em worth} increases, the
proportion of preferred projects also increases.

The rest of this paper describes how \fig{output} was generated.


\Section{JANE} A JANE programmer enters in models in  a
propositional rule-based language.
Internally, these rules are converted  into a directed graph with
the following BNF\footnote{In this article's BNF notation, !W ::==
X Y | Z! denotes that the structure $W$  contains either the
structures !X! and !Y!  or  the structure$Z$. !(X)*! denotes zero
or more repeats of !X!. !(X)+! denotes  one or more repeats of
!X!. ![X]! denotes that !X! is optional. Terminals   start with
lower case, or are quoted}:

\begin{code}
Graph  ::== Goal (Vertex)* (Edge)*
Goal   ::== Item
Vertex ::== Mix | Item
Item   ::== Type  Variable Value Label
Type   ::== action | fault | requirement
Mix    ::== CombineRules  Order
\end{code}
Note that each !Graph! has a special !Goal Item!. Conceptually,
JANE is a backward chainer that performs a recursive  descent from
the !Goal!.

In JANE, every vertex is either an !Item! where some !Value! is
assigned to some !Variable!; or a !Mix! vertex where influences
are combined. Each !Vertex! and !Edge! in JANE is augmented with a
!Cost! and a !Chances! weight and !Mix! vertices define how
!Cost!s and !Chances! are combined"

\begin{code}
Edge    ::== Vertex Label Vertex
Label   ::== Cost Chances
Cost    ::== null | 0.00 .. infinity
Chances ::== null | 0.00 .. 1.00
\end{code}
JANE's !Chances! define the extent to which a belief in one vertex
can propagate to another. !Cost!s  let an analyst  model the
common situation where some of the !Cost! of some procedure is
amortized by reusing its results many times. Hence, the {\em
first} time we use an !action! we incur that !Cost! but
afterwards, that !action! is free of charge.

The !Cost! and  !Chances! of  !Vertex!s are either provided by the
JANE programmer or computed at runtime via a traversal of the
edges. In JANE, this computation is defined by the !CombineRules!
at the !Mix! vertices. For each child of a !Mix! node, a
recursive descent is executed and the returned !Cost! and
!Chances! values are combined according to the !CombineRules!:

\begin{code}
CombineRules   ::== CostCombine
                  | ChancesCombine
CostCombine    ::== first(cost)
                  | sum(cost)
ChancesCombine ::== first(chances)
                  | sum(chances)
                  | negate
                  | product(chances)
\end{code}
!Negate! is used for negation. For example, when searching {\em X
if not A}, the !Chances! of {\em X} are {\em 1-Chances(A)}.
!Product(chances)! is used for {\em conjunctions}. For example,
when searching   {\em X if A !and! B !and! C}, the !Chances! of
{\em X} is the product of the chances of {\em A,B,C}. !First(X)!
is used for {\em simple disjunctive evidence}. For example, when
testing {\em X if A !or! B !or! C}, then the !Cost! and !Chances!
of {\em X} is taken from the first member of {\em A,B,C} that is
satisficed. !Sum(X)! is used for {\em summing  disjunctive
evidence}. For example, JANE supports a  special  operator called
!ors! that is used for implementing disjunctive summing (JANE also
supports several other novel operators, described in the next
section). !Ors! is like !or! except that when testing (e.g.) {\em
X if A !ors! B !ors! C}, all members of {\em A,B,C} will tested.
If at least one succeeds, the the !Cost! and !Chances! of {\em X}
is summed from the satisficed members of {\em A,B,C}.


!Sum(X)! and !Product(X)! can be combined to implement risk
mitigation effects. For example, suppose some {\em action5}
disables most of the contribution of  {\em fault3} on {\em
fault1}. In the JANE syntax, this can be coded as

\begin{code}
fault1 if fault2 @ 0.4
    ors (fault3@0.4 and not action5@0.9)
\end{code}
The !ors! operator  uses !Sum(chances)!; !and! uses
!Product(chances)!;
  and !not! uses !negate!. Hence,
if the !Chance!s of {\em fault2,fault3,action5} are all unity,
then of {\em action5} is false, the !Chance!s of {\em fault1} is
$0.4+(0.4*(1-0))=0.8$.
Alternatively, if {\em action5} is true,
the  !Chance!s of {\em fault1} is $0.4+(0.4*(1-0.9))= 0.44\ll
0.8$.  That is, using {\em action5}  significantly reduces the
!Chance!s of {\em fault1}.

\Section{JANE and Random Search}


JANE  supports two mechanisms for exploring the space of
possibilities within the CMM-2 encoding. Both are random search
tools. Firstly, when defining !Cost!s and !Chances!, the
programmer can supply a {\em range} and a {\em skew}. For example:

\begin{code}
goodUnitTesting and cost = 1 to +5
\end{code}
defines the {\em cost} of {\em goodUnitTesting} as being somewhere
in the range 1 to 5, with the mean skewed slightly towards 5
(denoted by the ``+''). During a simulation, the first time this
cost is accessed, it is assigned randomly according to the range
and skew. The assignment is cached so that all subsequent accesses
use the same randomly generated value. After each simulation, the
cache is cleared. After thousands of simulations, JANE can sample
the ``what-if'' behavior resulting from different
assignments within the range.

Secondly, JANE can randomized where it searches using the !Order!
of the !Mix! operators:

\begin{code}
Order      ::== Random | Left2Right
Left2Right ::== and  |  or | ors  |  any | not
Random     ::== rand | ror | rors | rany
\end{code}
In the case where the child-order represents some sequence that
must be preserved (e.g. ordering software processes) JANE
programmers can use !Left2Right! operators. However, repeatedly
searching a graph in the same order may miss important
interactions. In order not to miss such important interactions,
JANE supports several random search operators: !rand!, !ror!,
!rors!,  and !rany!.
For $op\in\{rand,ror\}$, JANE  tries to prove  all of {\em X op Y
op Z}, but does so in some randomly selected order.

!Rany! is best understood by comparison with !ror!. The expression
{\em X !ror! Y !ror! Z} is exited when any one of {\em X,Y,Z} is
satisficed.  In contrast,
JANE tries to prove one {\em or more} of {\em X !rany! Y !rany!
Z}, and does so in some randomly selected order. !Rany! is useful
when searching for subsets that contribute to some conclusion. For
example, the following JANE rule offers several essential features
of {\em
stableRequirements} plus several optional factors relating to
monitoring change in evolving projects. The essential features are
!rand!-ed together while the optional factors are !rany!-ed
together.

\begin{code}
{\scriptsize
stableRequirements   if effectiveReviews @ 0.3
    rand requirementsUsed @ 0.3
    rand sEteamParticipatesInPlanning @ 0.3
    rand documentedRequirements @ 0.3
    rand sQAactivities @ 0.3
    rand (reviewRequirementChanges @ 0.3
          rany softwareConfigurationManagement@0.3
          rany baselineChangesControlled @ 0.3
          rany workProductsIdentified @ 0.3
          rany softwareTracking @ 0.3).
}\end{code}

!Rors! is useful for specifying the high-level goals of the
system.   While  !rany! will search {\em some subset} of its
parameters,
  !rors!  will search {\em all} its parameters.
  For example, in this less-than-perfect world, it is unlikely we
can be {\em rich} and {\em happy} and {\em healthy}, but that
should not stop us from
trying; i.e.

\begin{code}
goal if rich rors happy rors healthy
\end{code}
 The operators  !rand!, !ror!, !rors! and !rany! have the same
satisficing criteria as !and!, !or!, !ors! and !any! respectively,
but the latter search left-to-right while the
former search in a randomly selected order. In terms of the
!CombineRules! described above, !And! and !rand! are  {\em
conjunctive} operators; !Or! and !ror! are   {\em simple
disjunctive} operators; and the rest are {\em summing disjunctive}
operators.

\Section{The CMM2 Model}

CMM2 written in JANE has 55 !Item!s with $!Value!\in\{t,f\}$. Of
the 55 !Item!s, 27 were identified as management actions that
could be changed by managers  (see \fig{actions}). The top-level
!Goal! was {\em goodProject}.

Within the model, !Chances! values   were  added to   79 edges of
the 150 edges in the model. While each of these values are based
on expert judgement, their precise value is subjective. Hence,
each such !Chances! value $X$ was altered to be a range

\begin{code}
chances = 0.7*X to 1.3*X
\end{code}
so the simulator could experiment with values nearby the stated
!Chances! value.


The full model is available via email from the authors.



\Section{Learning From JANE}\label{sec:worth}

One execution of JANE can be summarized by the !Cost! and
!Chances! of the top-level goal {\em goodProject},  the 79 values
assigned to the subjective edges, and the values assigned to the 27 manager's actions.
Random walking over JANE models can hence generate
an overwhelming amount of data (79+27+2=108 data points per
simulation, times the number of simulations).

Summarization of JANE output is performed by the TAR2 {\em
treatment learner}. A treatment learner outputs a set
of possible {\em treatments} ($\Delta$) using the process
described below. Each treatment is assessed by the {\em change} it
makes to the distribution of classes. A {\em good} treatment
increases the frequency of the {\em better classes} (where
``better'' is defined below).

Treatment learners input a set of classified examples ($Eg$). Each
example  is a conjunction of attribute ranges, plus one
classification $c$. This classification comes from a pre-defined
set of classes $C$; i.e. $c \in C$. Each classification $c$ is
associated with a numeric {\em score} i.e. some classifications
score better than others.

For the CMM model, after 2000 random walks, a wide range of
generated !Cost!s and !Change!s in {\em goodProject} were observed. These ranges were
sub-divided into high/low bands of  roughly the same size.
Combing high/low !Cost!/!Chances! yields four classes. These
classes were scored as follows:

\noindent {\bf Score=0}: {\em High cost}, {\em low chance}\newline
{\bf Score=1}: {\em Low cost},  {\em low chance} \newline {\bf
Score=2}: {\em High cost}, {\em high chance}\newline {\bf
Score=4}: {\em Low cost},  {\em high chance}.

That is, our preferred projects are cheap and highly likely while
expensive, low odds projects are to be avoided.

A treatment learner searches for {\em candidate}  attribute
ranges; i.e. ranges that are more common in the best
classification than in the not-so-best classification. In the CMM
domain, such a candidate is an attribute range that would tend to
drive the system into {\em low cost, high chance projects}.

A heuristic ranking $\delta$ is given to each candidate reflecting
(i)~how much more common is the candidate in the best class than
in some non-best-class $c$; and (ii)~how much better is the best
class than class $c$:

{\small
\[\delta=(score(best)-class(c))*
\left(\frac{|Eg \cap A=R \cap class=best|}{|Eg \cap A=R \cap
class=c|}\right)
\]}
where $A=R$ denotes an attribute $A$ with range $R$ and $|Eg \cap
X|$ denotes the size of the subset of examples that contain $X$.
%\[
%\delta_{A=R} = \sum_{c \in C} \begin{array}{l}\left\end{array}
%\]


Candidates with a large $\delta$ score are attribute ranges that
are far more frequent in the best class ({\em high chances, low
cost} projects) than in other classes. The {\em treatments}
$\Delta$ are all subsets of candidates with a ranking higher that
some user-specified $\delta$ threshold value $\delta_T$.

In reality, it is hard to change many aspects of a project and the
wise software engineer seeks the smallest number of changes with
the greatest impact. Assuming we seek three changes, there are 560
possible treatments of size 3 in our 16 candidates. The {\em best}
treatment is the one that offers the biggest improvement to the
class distributions seen  in the untreated data. To find the best
treatment, we compute following ratio for all 560 treatments:

{\small
\begin{equation}\label{eq:worth}
worth(\Delta_i)=\frac{\sum_{c\in C}|Eg \cap \Delta_i \cap
class=c|*score(c)}{baseline\; worth} \end{equation}} where {\em
baseline worth} is the {\em worth} seen in the untreated example
set:

{\small
\[{\small
baseline\;worth=\sum_{c\in C}|Eg \cap class=c|*score(c) }\]}

\section{Results}

\fig{cmm} shows the three best treatments ($\Delta_1, \Delta_2,
\Delta_3$)  found using this technique (and \fig{output} compared
the effects of these treatments to the untreated examples). Note
that the values of each attribute are reported using the tags {\em
no}, {\em lower}, {\em middle}, or {\em upper}. In treatment
learning, continuous attribute ranges are divided into N-discrete
bands based on percentile positions. For N=3, we can name the
bands {\em lower}, {\em middle}, {\em upper} for the lower,
middle, and upper  33\% percentile bands.

In \fig{cmm}, $\Delta_2$ and $\Delta_3$ are advising to lower the
cost of: \bd
\item[Using requirements:] This could be accomplished by (e.g.) by
sharing  them around the
development team in some searchable hypertext format
\item[Performing formal reviews at milestones:] This could be accomplished
by  (e.g.) using ultra-lightweight formal methods such as proposed
by Leveson~\cite{leveson91 }. \item[Performing good unit testing:]
This could be accomplished by (e.g.) hiring better test engineers.
\end{description}
\begin{figure}
{\small
\begin{description}
\item[$\Delta_1$:] {\em requirementsUsed.!Cost!=lower and
                    not periodicSoftware- Reviews and
                   formalReviewsAtMilestones.!Cost!=lower}
\item[$\Delta_2$:]
                   {\em requirementsUsed.!Cost!=lower and\newline
                   goodUnitTesting.!Cost!=middle and\newline
                   formalReviewsAtMilestones.!Cost!=lower}
\item[$\Delta_3$:]
                   {\em goodUnitTesting.!Cost!=lower and\newline
                   periodicSoftwareReviews.!Cost!=middle and\newline
                   formalReviewsAtMilestones.!Cost!=lower}
                   \end{description}
                   }
                   \caption{The three best treatments found in the CMM-2
model.}\label{fig:cmm}
                   \end{figure}
The value {\em no} refers to missing values; a report of {\em
X=no} is a recommendation {\em not} to use   $X$ as part of a
treatment. Such a negative  recommendation is seen in $\Delta_1$
which is advising  against {\em periodicSoftwareReviews} (plus
lowering the costs of using requirements and formal reviews at
milestones). Note that if {\em periodicSoftwareReviews} are
conducted, $\Delta_3$ is saying that there is no apparent need to
reduce the cost of such reviews.


\Section{DISCUSSION}
Space limitations prevent a  review of  work related to this research such as the goal-graphs of
Chung et.al.~\cite{chung00} (such a review is available from the authors).
This discussion section will hence focus on the major threat to the external validity of the above conclusions.

The knowledge source used in this
study (CMM2) is widely, but not universally,
accepted. Proponents of some other knowledge source could reject
the specifics of the above conclusions, but still use the general
technique; i.e. they could encode their preferred knowledge source in JANE and
repeat the above analysis.




\bibliographystyle{../../tex/latex8}
\bibliography{../../refs}
\end{document}
