Software process control allows managers to control software projects and keeping them within required specifications, whether for performance or cost or time to market. A repeated theme in literature~\cite{boehm99,boehm00a,boehm00b,chulani99,boehm88,boehm00} suggests that the most effective way of doing this is through using software engineering models that have been pre-calibrated to the particular software development environment. This assumes that there is an abundance of historical information that is available to tune these models. This however is a false assumption, as in reality there is a ``data drought" regarding project data~\cite{me07f}, causing uncertainty within the software models. So instead we developed a system that uses this uncertainty and mutates the models around the ranges of possible tunings using an AI search engine, while at the same time evaluating possible recommendations to control the software project. 

%What we observed is that we were able to produce results with a good deal of stability, and at the same time acheiving control over the project. In addition, we were able to use this system to do other studies relating to the subject of software process control. Based on this proof of concept design, future work has been initiated to develop this idea to tackle more issues by using search engines that are more appropriate, and furthering the concept to be able to tackle other software process control issues.

\section{Motivation}\label{sec:motive}
Process models have many purposes including estimating project parameters or conducting what-if queries to find better ways to organize a project. Most process models calibrate their internal settings using local data. Collecting this data is expensive, tedious, and often an incomplete process ~\cite{me06d}. For example, after 26 years of trying, Boehm et al.\cite{me07f} have only collected less than 200 sample projects for the COCOMO database. Also, even after two years of effort, only 7 records were added to a NASA-wide software cost metrics repository. For this reason we wanted to investigate whether or not we had the ability to:
\be
\item Make process decisions without historical data while achieving stability in those decisions.
\item Produce accurate estimates of different project properties (such as effort, defects, etc).
\item Move on to investigate other issues related to software process control.
\ee
We needed to determine which SE models to use for our studies, for which we chose the USC family of models. This includes the the COCOMO II.2000 effort model~\cite[p254-268]{boehm00b}, the COQUALMO defect model~\cite[p29-57]{boehm00b}, the Madachy risk model~\cite[284-291]{boehm00b}, and the COCOMO II based Months time of development (SCHEDULE) model~\cite{boehm00b}. These were chosen for their openness in terms of specification, their widespread adoption by such institutions like NASA and JPL, and for the wider availability of data on these models compared to other models in the software engineering industry. This is useful to us since it helps us determine variability in the parameters of these models, and therefore the space of tunings.

Having determined the models to be used, we attempted to evaluate whether we could fulfill the above stated goals. By using an AI search engine to explore the range of tunings of the selected process models, we were able to produce results with a good deal of stability, while at the same time achieving control over project estimates. We were also able to produce estimates with the ball park of figures produced with tuned models. In addition, we were able to use this system to do other studies relating to the subject of software process control. %Note that with a search based algorithm we are able to avoid getting stuck at local optima.

Based on our results and empirical data, we present the following statement of thesis.

%Based on this proof of concept design, future work has been initiated to develop this idea to tackle more issues by using AI search engines that are more appropriate, and furthering the concept to be able to tackle other software process control issues. 
%Moving along to the selection of our methodology, we decided to work with a stochastic search based algorithm to evaluate and score candidate policies. By doing this we are able to exploit the advantages of such algorithms, one of which is the ability of such algorithms to "jump out of" local maxima/minima. To provide a baseline for future work, Simulated Annealing\ref{sec:sa} was used as the stocastic search algorithm. This system is described further later on in this document.

\section{Statement of Thesis}\label{sec:statement}
Variability in software process model output arises from (a)~uncertainty in model inputs relating to the project and (b)~uncertainty in the tuning of internal parameters that control the process model outputs. We find that, for USC process models such as COCOMO and COQUALMO, the model ouputs are less dependent on the uncertainty in the internal parameters of the model as compared to the dependence on the uncertainty of the project parameters. We are hence able to control the variance of the process models by setting project parameters rather than resorting to calibrating the models.

\section{Contribution of this Work}\label{sec:contrib}
The area of search based software engineering is one that is growing at the moment at a quick pace~\cite{harman05}, and this work further contributes to this field. We introduce a novel method of evaluating and controlling the software process, using four of the USC family models in conjunction with an AI search engine, without the need for local tuning.
From a business perspective, this result means that certain process models can be used for decision making in one of two ways:
\be
\item
Either constrain the tuning variance using historical data;
\item
Or constrain the project choices using an AI search engine like STAR, presented in \tion{staralgo}.
\ee
Note that this second method avoids a lengthy and expensive data collection phase prior to decision making. This result is of tremendous practical benefit since it is often very difficult to find relevant data within a single organization to precisely tune all the internal parameters inside a process model. 
This thesis, in addition to the above, also contributes several case studies and experiments. The data of these studies is publicly available, along with the tool developed and its source code, for third parties to attempt to regenerate the results presented. Note that this work has also been used as a proof of concept for further work. 

\subsection{Publications from this work}

This work has generated several publications:

\bi
\item  {\bf ICSE'09}: {\em How to avoid drastic software process change}. \cite{menzies09}
\item {\bf ASE'07}:  {\em The business case for automated software engineering}. \cite{me07f}
\item {\bf ASE'07 workshop}: {\em On the value of stochastic abduction}.
In {\em International Workshop on Living with Uncertainty }. \cite{me07g}
 \item {\bf ICSP'08}: {\em Accurate estimates without calibration} (in the International Conference on Software Process). \cite{me08c}
\ei

In addition to the above publications, there are plans to generate several others directly derived from this work.

\section{Relation to Other Work}
This thesis discusses the design of an AI decision support system called STAR.
STAR's design has inspired other work by other WVU graduate students:
For example, the author  assisted another WVU graduate student, Mr. Steve Williams, in his work building a LISP version of STAR. 

In order to assign appropriate credit to the work reported in this document, it should be noted that:
\bi
\item For all the STAR results reported in this thesis,
El-Rawas was the lead analyst/programmer.
\item  For all the SEESAW results reported in this thesis,
Williams was the lead analyst/programmer while El-Rawas served as programmer and assistant.
\ei

\section{Document Description}\label{sec:description}
The rest of this document is organized as follows. Chapter 2 will present previous work that is relevant to this thesis. This is followed by a detailed description and analysis of STAR in chapter 3, accompanied several internal studies to verify its use. Chapter 4 presents the NASA case studies and the case for the use of automated software engineering tools. Chapter 5 presents studies done in conjunction with experts in the field. Chapters 6 and 7 are case studies into two software engineering practices: Drastic change and ``better, faster, cheaper''. The document is concluded with some notes on the results, external validity and future work that is already ongoing. The appendix contains the code listing and how to obtain the tool presented here.
