%\documentclass[journal]{../one/IEEEtran}
%\usepackage{alltt}
%\usepackage{syntax}
%\usepackage{bar}
%\usepackage{wrapfig}
%\usepackage{rotating}
%\usepackage{cite}      % Written by Donald Arseneau
%                        % V1.6 and later of IEEEtran pre-defines the format
%                        % of the cite.sty package \cite{} output to follow
%                        % that of IEEE. Loading the cite package will
%                        % result in citation numbers being automatically
%                        % sorted and properly "ranged". i.e.,
%                        % [1], [9], [2], [7], [5], [6]
%                        % (without using cite.sty)
%                        % will become:
%                        % [1], [2], [5]--[7], [9] (using cite.sty)
%                        % cite.sty's \cite will automatically add leading
%                        % space, if needed. Use cite.sty's no adjust option
%                        % (cite.sty V3.8 and later) if you want to turn this
%                        % off. cite.sty is already installed on most LaTeX
%                        % systems. The latest version can be obtained at:
%                        % http://www.ctan.org/tex-archive/macros/latex/contrib/supported/cite/
%
%\usepackage{graphicx}  % Written by David Carlisle and Sebastian Rahtz
%                        % Required if you want graphics, photos, etc.
%                        % graphicx.sty is already installed on most LaTeX
%                        % systems. The latest version and documentation can
%                        % be obtained at:
%                        % http://www.ctan.org/tex-archive/macros/latex/required/graphics/
%                        % Another good source of documentation is "Using
%                        % Imported Graphics in LaTeX2e" by Keith Reckdahl
%                        % which can be found as esplatex.ps and epslatex.pdf
%                        % at: http://www.ctan.org/tex-archive/info/
%
%\usepackage{subfigure} % Written by Steven Douglas Cochran
%                        % This package makes it easy to put sub figures
%                        % in your figures. i.e., "figure 1a and 1b"
%                        % Docs are in "Using Imported Graphics in LaTeX2e"
%                        % by Keith Reckdahl which also documents the graphicx
%                        % package (see above). sub figure.sty is already
%                        % installed on most LaTeX systems. The latest version
%                        % and documentation can be obtained at:
%                        % http://www.ctan.org/tex-archive/macros/latex/contrib/supported/subfigure/
%
%\usepackage{url}       % Written by Donald Arseneau
%                        % Provides better support for handling and breaking
%                        % URLs. url.sty is already installed on most LaTeX
%                        % systems. The latest version can be obtained at:
%                        % http://www.ctan.org/tex-archive/macros/latex/contrib/other/misc/
%                        % Read the url.sty source comments for usage information.
%
%%\usepackage{stfloats}  % Written by Sigitas Tolusis
%                        % Gives LaTeX2e the ability to do double column
%                        % floats at the bottom of the page as well as the top.
%                        % (e.g., "\begin{figure*}[!b]" is not normally
%                        % possible in LaTeX2e). This is an invasive package
%                        % which rewrites many portions of the LaTeX2e output
%                        % routines. It may not work with other packages that
%                        % modify the LaTeX2e output routine and/or with other
%                        % versions of LaTeX. The latest version and
%                        % documentation can be obtained at:
%                        % http://www.ctan.org/tex-archive/macros/latex/contrib/supported/sttools/
%                        % Documentation is contained in the stfloats.sty
%                        % comments as well as in the presfull.pdf file.
%                        % Do not use the stfloats baseline float ability as
%                        % IEEE does not allow \baselineskip to stretch.
%                        % Authors submitting work to the IEEE should note
%                        % that IEEE rarely uses double column equations and
%                        % that authors should try to avoid such use.
%                        % Do not be tempted to use the cited.sty or
%                        % midfloat.sty package (by the same author) as IEEE
%                        % does not format its papers in such ways.
%
%\usepackage{amsmath}   % From the American Mathematical Society
%                        % A popular package that provides many helpful commands
%                        % for dealing with mathematics. Note that the AMSmath
%                        % package sets \interdisplaylinepenalty to 10000 thus
%                        % preventing page breaks from occurring within multiline
%                        % equations. Use:
%%\interdisplaylinepenalty=2500
%                        % after loading amsmath to restore such page breaks
%                        % as IEEEtran.cls normally does. amsmath.sty is already
%                        % installed on most LaTeX systems. The latest version
%                        % and documentation can be obtained at:
%                        % http://www.ctan.org/tex-archive/macros/latex/required/amslatex/math/
%
%
%
%% Other popular packages for formatting tables and equations include:
%
%\usepackage{array}
%% Change the following for any hyphenation issues.  These are just examples. (Not actually used yet.)
%\hyphenation{op-tical net-works semi-conduc-tor}
%\input{../jim/tricks.sty}
%\input{../tim/tricks.sty}
%\input{../jeremy/tricks.sty}
%\begin{document}
%%
%% paper title
%\title{Exploiting Models in Software Engineering}
%%
%%
%% author names and IEEE memberships
%% note positions of commas and non breaking spaces ( ~ ) LaTeX will not break
%% a structure at a ~ so this keeps an author's name from being broken across
%% two lines.
%% use \thanks{} to gain access to the first footnote area
%% a separate \thanks must be used for each paragraph as LaTeX2e's \thanks
%% was not built to handle multiple paragraphs
%\author{Tim Menzies}
%% <-this % stops a space\thanks{Contact information: kiperjd@muohio.edu, timm@cs.pdx.edu, jegreen@cs.pdx.edu .}}
%% note the % following the last \IEEEmembership and also the first \thanks - 
%% these prevent an unwanted space from occurring between the last author name
%% and the end of the author line. i.e., if you had this:
%% 
%% \author{....lastname \thanks{...} \thanks{...} }
%%                     ^------------^------------^----Do not want these spaces!
%%
%% a space would be appended to the last name and could cause every name on that
%% line to be shifted left slightly. This is one of those "LaTeX things". For
%% instance, "A\textbf{} \textbf{}B" will typeset as "A B" not "AB". If you want
%% "AB" then you have to do: "A\textbf{}\textbf{}B"
%% \thanks is no different in this regard, so shield the last } of each \thanks
%% that ends a line with a % and do not let a space in before the next \thanks.
%% Spaces after \IEEEmembership other than the last one are OK (and needed) as
%% you are supposed to have spaces between the names. For what it is worth,
%% this is a minor point as most people would not even notice if the said evil
%% space somehow managed to creep in.
%%
%% The paper headers
%%\markboth{Journal of \LaTeX\ Class Files,~Vol.~1, No.~11,~November~2002}{Shell \MakeLowercase{\textit{et al.}}: Bare Demo of IEEEtran.cls for Journals}
%% The only time the second header will appear is for the odd numbered pages
%% after the title page when using the two side option.
%% 
%% *** Note that you probably will NOT want to include the author's name in ***
%% *** the headers of peer review papers.                                   ***
%
%% If you want to put a publisher's ID mark on the page
%% (can leave text blank if you just want to see how the
%% text height on the first page will be reduced by IEEE)
%%\pubid{0000--0000/00\$00.00~\copyright~2002 IEEE}
%
%% use only for invited papers
%%\specialpapernotice{(Invited Paper)}
%
%% make the title area
%
%%\footrulewidth 0.0pt\headrulewidth 0.0pt
%%\lhead{}\rhead{}\chead{}\rfoot{}\lfoot{}\cfoot{}
%%
%%\rhead{\includegraphics[height=0.5in]{ASEbiglogo.eps}}
%%\lhead{{\footnotesize { Submitted to the 16th IEEE International
%%Conference on Automated Software Engineering
%%                                                                          Nov
%%20-22, San Diego, USA.\newline
%%\url{http://sigart.acm.org/Conferences/ase/}\newline WP ref:
%%01/choices/ \today }}} \headrulewidth 0.4pt
%
%%\documentstyle[alltt,icse2000,twocolumn,url,bar,epsfig,times,alltt,array,amssymb,graphicx,ifthen,color]{article}
%
%% Changes:
%%  Page limit: 10 pages
%%  Abstract limit: 200 words
%%  Contact information is now gall@infosys.tuwien.ac.at
%%  URL for author kit changed: needs to be filled in.
%%  All other font and sizing information is the same:
%%    check whether it still applies.
%
%%\makeatletter
%%\def\fnum@figure{\sffamily\figurename~\thefigure}
%%\def\fnum@table{\sffamily\tablename~\thetable}
%%\makeatother
%
\newenvironment{code}{\begin{alltt}\small}{\normalsize\end{alltt}}
%
\newenvironment{ccode}{\begin{code}\begin{center}}{\end{center}\end{code}}
%
%
%
%
%
%\newcommand{\DO}  {{\sffamily \{}}
%\newcommand{\FOR}{{\underline {\sffamily for}}}
%\newcommand{\DONE}{ {\sffamily \}}}
%\newcommand{\IF}{{\underline {\sffamily if}}}
%\newcommand{\FI}{{\underline {\sffamily fi}}}
%\newcommand{\THEN}{{\underline {\sffamily then}}}
%\newcommand{\ELSE}{{\underline {\sffamily else}}}
%\newcommand{\WHEN}{{\underline {\sffamily when}}}
%\newcommand{\UNTIL}{{\underline {\sffamily until}}}
%\newcommand{\AND}{{\underline {\sffamily and}}}
%\newcommand{\TO}{{\underline {\sffamily to}}}
%\newcommand{\IS}{ {\sffamily $\leftarrow$}}
%
%\newcommand{\NIL}{ {\sffamily $\emptyset$}}
%\newcommand{\REPEAT}{{\underline {\sffamily repeat}}}
%\newcommand{\RETURN}{{\underline {\sffamily return}}}
%
%\shortverb{!}
%
%\copyrightspace
%
% If you want to print drafts of the paper with a draft
% notice in the copyright space, comment out the \copyrightspace
% line above and include the \submitspace line below instead.
%
%\submitspace{Draft of paper submitted to ASE 2000.
%\\WP:b/00/ase/whatif(\today).}


% Page number on the initial page can be omitted in both the review and
% final submission (and should be removed in the final submission).  The
% line below does that.


% In the review submission, page numbers should appear (they can be omitted
%  from the first page).  The page style command below puts them in.
% In the final submission of accepted papers, page numbers should be
%  omitted; remove or comment out the page style line below to omit them.


% Use \section* instead of \section to suppress numbering for
% the abstract, acknowledgments, and references.


%program too simple? XXX space too simple? no- decision tree

\subsection{Studying the CMM}\label{sec:cmm}

The previous  studied explored a numeric
model where all the influences were precisely specified.
This second case study takes a numeric model and adds a large
degree of uncertainty in the numerics.
This second study shows that, even in presence of large degrees of uncertainty,
TAR2 can still find useful treatments.

An important feature of this second study is that it analyzes a class of models that
can defeat standard methods. The model contains dozens of if-then rules; i.e.
it is neither linear nor continuous: small changes in the environment can
lead to ``cliffs'' where the model behavior changes abruptly. Also, the model
contains nondeterministic choices (see the $rany$ operator, discussed below)
and so its behavior can be highly noisy.

This study uses a rule-based model of the costs
and 
benefits model of CMM level 2 (hereafter,
CMM2)~\cite[p125-191]{paulk95}. We elected to study CMM2 since, in our
experience, many organizations can achieve at least this level.
CMM2 is less concerned with issues of (e.g.) which design pattern
to apply, than with what overall project structure should be
implemented. Improving CMM2-style decisions is important since in
early software life cycle, many CMM2-style decisions affect the
resource allocation for the rest of the project.

CMM2 was encoded using the JANE propositional rule-based language~\cite{me01e}. 
JANE's rules take the form \mbox{$Goal\; if\; SubGoals$} such as the one
shown in \fig{jane}.
\begin{figure}[!h]
\renewcommand{\baselinestretch}{1}
\begin{code}
{\scriptsize
stableRequirements  
    if   effectiveReviews            
    and requirementsUsed            
    and sEteamParticipatesInPlanning 
    and documentedRequirements       
    and sQAactivities                
    and (reviewRequirementChanges    
          rany softwareConfigurationManagement 
          rany baselineChangesControlled      
          rany workProductsIdentified        
          rany softwareTracking
        ).
}\end{code}
\caption{Part of CMM2, encoded in the JANE language.}\label{fig:jane}
\end{figure}

JANE is a backward chaining language:
to prove a $Goal$, JANE tries to find rules that prove each of  the $SubGoals$.
Each $SubGoal$ contributes some $Cost$  and $Chances$
to the $Goal$. 
JANE's $Chances$ define the extent to which a belief in one vertex
can propagate to another. $Cost$s  let an analyst  model the
common situation where some of the $Cost$ of some procedure is
amortized by reusing its results many times. Hence, the {\em
first} time we use a proposition, we incur its $Cost$ but
afterwards, that proposition is free of charge.

The $Cost$ and  $Chances$ of  a proposition
are  either provided by the
JANE programmer or computed at runtime via a traversal of the
rules:
\bi
\item
When searching {\em X
if not A}, the $Chances$ of {\em X} are {\em 1-Chances(A)}
and $Cost(X)=Cost(A)$.
\item
When searching   {\em X if A $and$ B $and$ C}, the $Chances$ and
$Costs$ of
{\em X} are (respectively)
the product of the chances and the sum of the costs
of {\em A,B,C}. 
\item
When
searching {\em X if A $or$ B $or$ C}, then the $Cost$ and $Chances$
of {\em X} are taken from the first member of {\em A,B,C} that is
satisfied.  
\ei

These
$and, or, not$ operators can be
insufficient to capture the  decision making of business users.
For examples,
in our experience,
business users select CMM2 options, often in a somewhat 
arbitrary manner.
To model this, JANE  includes a  $rany$ operator
(short for ``random any''):
\bi
\item
The $rany$ operator is  like $or$ except that (e.g.) {\em
X if A $rany$ B $rany$ C} succeeds if some
random number of {\em A,B,C}  (greater than one) succeeds.
Unlike $and,or$ which explore their operands in a left-to-right
order, $rany$ explores its $SubGoals$ is a random order.
If at least one succeeds, then the $Cost$ 
and $Chances$ of {\em X}
is the sum and product (respectively)
of the $Cost$ and $Chances$ of
the satisfied members of {\em A,B,C}.
\ei
\begin{figure}[!b]
\renewcommand{\baselinestretch}{1}
{\small\begin{tabular}{|p{1.5in}p{1.5in}|}\hline
\begin{center}
{\em baselineAudits, baselineChangesControlled,
     changeRequestsHandled,
     changesCommunicated,
     configurationItemStatusRecorded,
     deviationsDocumented,
     documentedDevelopmentPlan,
     documentedProjectPlan,
     earlyPlanning,
     \underline{formalReviewsAtMilestones}, \underline{goodUnitTesting},
     identifiedWorkProducts,
     \underline{periodicSoftwareReviews}
     }\end{center} &  \begin{center}{\em
     planRevised, requirementsReview,
     \underline{requirementsUsed},
     reviewRequirementChanges,
     risksTracked,
     SCMplan,
     SCMplanUsed,
     SElifeCycleDefined,
     SEteamParticipatesInPlanning,
     SEteamParticipatesOnProposal,
     SQAauditsProducts,
     SQAplan,
    SQAplanUsed,
     SQAreviewActivities,
    workProductsIdentified} \end{center}\\\hline
     \end{tabular}}
\caption{Management actions in the CMM2 model.    SQA= software
quality     assurance and SCM= software configuration management)}\label{fig:actions}
     \end{figure}


\begin{figure*}
\renewcommand{\baselinestretch}{1}
\begin{minipage}{.75\linewidth}
{\small
\begin{center}
\begin{tabular}{cccc}
\begin{barenv}
\setstyle{\sffamily} \setwidth{8}
  \setyaxis{0}{75}{10}
\sethspace{1} \setnumberpos{down} \bar{5}{1} \bar{16}{2}
\bar{5}{4} \bar{74}{8}
\end{barenv}

&

\begin{barenv}
\setstyle{\sffamily} \setwidth{8}
  \setyaxis{0}{75}{10}
\sethspace{1} \setnumberpos{down} \bar{7}{1} \bar{13}{2}
\bar{19}{4} \bar{60}{8}
\end{barenv}

&

\begin{barenv}
\setstyle{\sffamily} \setwidth{8}
  \setyaxis{0}{75}{10}
\sethspace{1} \setnumberpos{down} \bar{5}{1} \bar{14}{2}
\bar{26}{4} \bar{55}{8}
\end{barenv}
&

\begin{barenv}
\setstyle{\sffamily} \setwidth{8}
  \setyaxis{0}{75}{10}
\sethspace{1} \setnumberpos{down} \bar{14}{1} \bar{22}{2}
\bar{30}{4} \bar{24}{8}
\end{barenv}


\\~\\
$T_1$  &
$T_2$ & 
$T_3$ & 
baseline    
\\
  worth=1.44 &
  worth=1.31 &  worth=1.28
& current worth = 1
\end{tabular}
\end{center}
}\end{minipage}\hfill
\begin{minipage}{.25\linewidth}
\begin{raggedleft}
{\scriptsize

\begin{tabular}{|cp{1in}|}\hline
\multicolumn{2}{|p{1.5in}|}{~\newline\underline{{\bf KEY:}}
\newline Top-to-bottom = least

desirable to most desirable.\newline~\newline}\\
\protect\legend1{~}&\begin{raggedright}= high cost, low chances;
i.e. a very bad software \newline project\end{raggedright}
\\
\protect\legend2{~}&\begin{raggedright}= low cost, low
chances\end{raggedright}\\
\protect\legend4{~}&\begin{raggedright}= high cost, high
chances\end{raggedright}\\
\protect\legend8{~}&\begin{raggedright}= low cost, high chances;
i.e. a good software\newline project\end{raggedright}\\\hline
\end{tabular}}
\end{raggedleft}
\end{minipage}
\caption{ Ratios of different software project types seen in four
situations. }\label{fig:output}
\end{figure*}


$Rany$ is useful
when searching for subsets that contribute to some conclusion. For
example, the JANE rule in \fig{jane}
offers several essential features
of {\em
stableRequirements} plus several optional factors relating to
monitoring change in evolving projects -- the essential features are
$and$-ed together while the optional factors are $rany$-ed
together.

\fig{jane} includes 11 propositions.
Our model of CMM2, written in JANE, has 55 propositions 
($range=\{t,f\}$).
Of
those 55 propositions,
27 were identified by our users
as actions that
could be changed by managers  (see \fig{actions}). 


Apart from $rany$,
JANE  supports one other mechanisms for exploring the space of
possibilities within CMM2. 
When defining $Cost$s and $Chances$, the
programmer can supply a {\em range} and a {\em skew}. For example:

\begin{code}
goodUnitTesting and cost = 1 to +5
\end{code}
defines the {\em cost} of {\em goodUnitTesting} as being somewhere
in the range 1 to 5, with the mean skewed slightly towards 5
(denoted by the ``+''). 

Similarly, 
while all the $Chances$ values were based
on expert judgment, their precise value is subjective. Hence,
each such $Chances$ value $X$ was altered to be a range

\begin{code}
chances = 0.7*X to 1.3*X
\end{code}

During a simulation, the {\em first} time a $Cost$ or $Chance$
is accessed, it is assigned randomly according to the range
and skew. The assignment is cached so that all subsequent accesses
use the same randomly generated value. After each simulation, the
cache is cleared. After thousands of simulations, JANE can sample
the ``what-if'' behavior resulting from different
assignments within the range and many different $rany$ choices.

Data from 2000 simulations was passed from the CMM2 model
to TAR2.
Each simulation was classified into one of four classes:

\bi
\item
 {\em class=0}: High cost, low chance;
\item
{\em class=1}:  Low cost, low chance;
\item
{\em class=2}: High cost,  high chance;
\item
{\em class=3}: 
Low cost,  high chance.
\ei

That is, our preferred projects are cheap and highly likely while
expensive, low odds projects are to be avoided.


\fig{output} shows three sets of actions  learned by TAR2.
The right-hand-side histogram shows the baseline
distributions seen in the 2000 simulations.
The other histograms  show how those
ratios change after applying the  treatments learned by TAR2;
The {\em worth} of each option is a reflection of the
proportion of good and bad projects, compared to the baseline,
i.e.
($worth(baseline)=1$). Note that as {\em worth} increases, the
proportion of preferred projects also increases.



\fig{cmm} shows the three best treatments ($T_1, T_2,
T_3$)  found using this technique (and \fig{output} compared
the effects of these treatments to the untreated examples). Note
that the values of each attribute are reported using the tags {\em
no}, {\em lower}, {\em middle}, or {\em upper}. In treatment
learning, continuous attribute ranges are divided into N-discrete
bands based on percentile positions. For N=3, we can name the
bands {\em lower}, {\em middle}, {\em upper} for the lower,
middle, and upper  33\% percentile bands.

In \fig{cmm}, the treatments
are advising to lower the
cost of: \bi
\item {\em Using requirements:} This could be accomplished by (e.g.) 
sharing  them around the
development team in some search-able hypertext format
\item {\em Performing formal reviews at milestones:} This could be accomplished
by  (e.g.) using ultra-lightweight formal methods such as proposed
by Leveson~\cite{leveson91 }. \item {\em Performing good unit testing:}
This could be accomplished by (e.g.) hiring better test engineers.
\ei
\begin{figure}[!t]
\renewcommand{\baselinestretch}{1}
{\small
\begin{description}
\item[$T_1$:] {\em requirementsUsed.Cost=lower and\newline
                    not periodicSoftware-Reviews and\newline
                   formalReviewsAtMilestones.Cost=lower}
\item[$T_2$:]
                   {\em requirementsUsed.Cost=lower and\newline
                   goodUnitTesting.Cost=middle and\newline
                   formalReviewsAtMilestones.Cost=lower}
\item[$T_3$:]
                   {\em goodUnitTesting.Cost=lower and\newline
                   periodicSoftwareReviews.Cost=middle and\newline
                   formalReviewsAtMilestones.Cost=lower}
                   \end{description}
                   }
                   \caption{The three best treatments found in the CMM2
model.
}\label{fig:cmm}
                   \end{figure}

An interesting feature of \fig{cmm} is what is {\em missing}:
\bi
\item
None of the treatments proposed adjusting the
$Chances$ of any action. In this study, changing $Cost$ will
suffice.
\item
Of the 27 actions
listed in in \fig{actions}, only the four underlined actions
appear in the top three treatments. That is, management
commitment to undertake  27-4=23 of the actions is
less useful than changing  on
     $formalReviewsAtMilestones$, $goodUnitTesting$,
     $periodicSoftwareReviews$, and
     $requirementsUsed$
\item
   The value {\em not} in 
$T_1$
is a recommendation 
 against {\em periodicSoftwareReviews} (plus
lowering the costs of using requirements and formal reviews at
milestones). Note that if {\em periodicSoftwareReviews} are
conducted, $T_3$ is saying that there is no apparent need to
reduce the cost of such reviews.
\ei

More generally, in a result consistent with the prior studies,
despite the uncertainties introduced by $rany$
and the $cost/chances$ ranges,
TAR2 found
a small number of CMM2 process options that  
have a significant impact on the
project. 

Note that the conclusions of \fig{cmm} are not general to all
software projects. The $Chances$ values used in this study came
from some local domain knowledge about the likelihood that process
change $A$ will effect process change $B$.  
The $Cost$ values were domain-specific as well.
In other organizations,
with different work practices and staff, those $Chances$ and $Cost$
values could be
very different.

% study use
%XXX jim- some phrase about external validity. only good for the
%numbers we used in this study. blah blah blah

%\bibliographystyle{../one/IEEEtran}
%\bibliography{../one/refs,../one/miamiRefs,../jim/refs,../tim/refs,../jeremy/refs}
%\end{document}
