\section{MyModel}

MyModel\footnote{Full source code is located in section \ref{src-mymodel}} was used as the example model in chapter \ref{nova-programmer-interface}.

\section{\xomo}

In this section, we will describe \xomo{}, which is a software model that combines combines
Effort\index{Effort} and Months\index{Months} from COCOMO II\cite{boehm00b}\index{COCOMO II},
Defects\index{Defects} from COQUALMO\cite{boehm00b}\index{COQUALMO},
and Threats\index{Threats} from Madachy\cite{madachy94}.

\subsection{Effort \cite{boehm00b}\index{Effort}}\label{subsec:xomo-effort}
The COCOMO II\index{COCOMO II} model~\cite{boehm00b} is the successor to the original COCOMO 81\index{COCOMO 81} model~\cite{boehm81} which was published by Barry Boehm.
Both models use similar equations for calculating Effort.
This form is shown in \eq{general-cocomo-effort}, where $c_{i}$ is the sum of the cost drivers and $A$ is domain specific.

{\begin{equation}\label{eq:general-cocomo-effort}
Effort = A*(c_{i})(Size)^{Exponent}
\end{equation}}

The COCOMO II\index{COCOMO II} and COCOMO 81\index{COCOMO 81} differ in how $Size$ and $Exponent$ are calculated and what attributes are in $c_{i}$.
COCOMO 81 calculated $Size$ using the deliverable source lines of code (DSI).
In COCOMO II, three components make up size: source lines of code (SLOC), functional points, and object points.
Unlike DSI, SLOC is counted using logical language statements specified in the Software Engineering Institute guidelines~\cite{cmu-92tr020}.
$Exponent$ in COCOMO 81 was one of three constants based upon the Development Mode~\cite{boehm81}.
This has been replaced by an expression that uses COCOMO II's scale factors.
The more verbose effort calculation for COCOMO II can be seen in \eq{cocomoii-effort}.
D. Reifer, B. Boehm, and S. Chulani have created a guide~\cite{usccse98-516} to convert projects from COCOMO 81 to COCOMO II.
\xomo{} and this thesis only uses COCOMO II.  Because of this, from this point on, COCOMO refers to COCOMO II.

\input{tables/cocomoII-attribute-descriptions}

\input{tables/cocomoII-coefficient-values}

\input{tables/cocomoII-rating-ranges}

\glossary{name={Person Month},description={Amount of work that can be accomplished by an average person in one month.  The duration of a month is 152 working hours.}}

COCOMO's Effort\index{effort} estimates in units of Person Months\index{person month}.
A person month is the amount of work that can be accomplished by an average person in one month.
The duration of a month is 152 working hours.
Attributes for COCOMO are divided into two groups as can be seen in \tab{cocomoii-coefficients}.
The \textit{scale factors}\index{scale factors} ($SF_i$) effect effort exponentially,
while \textit{effort multipliers}\index{effort multiplier} ($EM_j$) effect effort linearly.
{\small\begin{equation}\label{eq:cocomoii-effort}
PM=a*\left(\prod_{j=1}^{17}EM_j\right)*\left(KSLOC^{\left(b+0.01*\sum_{i=1}^{5}SF_i\right)}\right)
\end{equation}}
$KSLOC$ is thousand source lines of code ($SLOC/1000$).
Both $a$\index{effort!a} and $b$\index{effort!b} are domain-specific parameters that need calibrated per each organization.
In the \ac{NASA} data\cite{nasa-data} we were using, we found the following ranges for $a$ and $b$:
\begin{equation}\label{eq:a-range}
\left( 3.72 \le a \le 9.18 \right)
\end{equation}
\begin{equation}\label{eq:b-range}
\left(0.88 \le b \le 1.09\right)
\end{equation}

Scale factors affect the effort when their values are not xh (6).
Knowing this, scale factors tunings can be modeled linearly around the point \{x,y\} = \{6,0\}:
\begin{equation}\label{eq:sf-linear-tuning-model}\small
 \forall x \in \{1..6\}\; SF_i=m_{SF_i}(x - 6)
\end{equation}
$m_{SF_i}$ denotes the effect of scale factor $SF_i$ on \textit{effort}.
Based on prior work of 161 projects~\cite{boehm00a}, the range tunings for scale factors (prec, flex, resl, team, pmat) are
\begin{equation}\label{eq:sf-linear-tuning-model-slope-range}\small
\left(-1.560 \le m_{SF_i} \le -1.014\right)
\end{equation}

Analogously, effort multipliers can be modeled linearly as well.
Effort multipliers affect the effort when their values are not nominal (3).
By using the point \{x,y\} = \{3,1\}:
\begin{equation}\label{eq:em-linear-tuning-model}\small
\begin{array}{l}
\forall x \in \{1..6\}\; EM_j=m_{EM_j}(x-3) + 1
\end{array}
\end{equation}
$m_{EM_j}$ denotes the effect of effort multiplier $EM_j$ on \textit{effort}.
Unlike scale factors, some multipliers increase effort, while others decrease effort.
Therefore, the milliliters can be broken into two group:
\begin{itemize}
\item
The \textit{positive effort EM} features, with slopes $m^+_{EM_j}$, that are proportional to effort.
These features are:
cplx,
data,
docu,
pvol,
rely,
ruse,
stor, and
time.
\item
The \textit{negative effort EM} features, with slopes $m^-_{EM_j}$,
are inversely proportional to effort.
These features are
acap,
apex,
ltex,
pcap,
pcon,
plex,
sced ,
site,
tool.
\end{itemize}
Just as with scale factors, prior work~\cite{boehm00a}, the range of tuning factors are
\begin{equation}\label{eq:em-linear-tuning-model-slope-range-positive}\small
\left(0.073 \le m^+_{EM_j} \le 0.210\right)
\end{equation}
\begin{equation}\label{eq:em-linear-tuning-model-slope-range-negative}\small
\left(-0.178 \le m^-_{EM_j} \le -0.078\right)
\end{equation}

\textbf{Note:}
The above tuning ranges for the slopes were obtained by finding the average slope for each COCOMO attribute
for both effort multipliers and scale factors over the range of values of that attribute.


\subsection{Schedule \cite{boehm00a}}
Similar to effort\index{effort}, Schedule\index{schedule} estimates development time in terms of calendar months.
As can be seen in \eq{cocomoii-schedule}, schedule is dependent on effort.
Schedule uses the same attributes, values, calibrated coefficients,
and even a slightly modified $PM$ calculated from the equation used in effort.

{\small\begin{equation}\label{eq:cocomoii-schedule}
\begin{array}{c}
F=d+0.2*\left(0.01*(\sum_{i=1}^{5}SF_i)\right)\\
TDEV=\left[c*(PM_{NS})^{F}\right]*\frac{SCED\%}{100}
\end{array}
\end{equation}}

$PM_{NS}$ is nearly identical to $PM$ from the effort calculation.
The only difference is that $PM_{NS}$ always uses nominal SCED.
$SF_i$ are the same \textit{scale factors}\index{scale factors} specified in \tab{cocomoii-coefficients}.
Just as in effort, schedule has its own domain-specific parameters: $c$\index{schedule!c} and $d$\index{schedule!d}.
\tab{cocomoii-schedule-scedpercent} shows the values of $SCED\%$, which is the percentage of SCED to the nominal SCED.
$F$ is referred to as the scaling exponent of schedule and it simply the exponent portion of $TDEV$.
$TDEV$ is the time to develop in months.

\input{tables/cocomoII-scedule-scedpercent-values}

\subsection{Defects \cite{boehm00b}}
The COQUALMO\index{COQUALMO} model can be used to estimate the total number of defects per thousand lines of code (KSLOC).
This is accomplished by combining the following two submodels:
\begin{itemize}
\item
The \textit{defect introduction} model is similar to
\eq{cocomoii-effort}; i.e. \tab{cocomoii-coefficients}'s effort multipliers and scale factors
map to estimates about number of defects.
\item
The \textit{defect removal} model estimates how different strategies (peer review, execution-based
testing, and automatic analysis) decrease the introduction of defects.
\end{itemize}

The two mentioned submodules are calculated for three stages of development:
requirements, design, coding.  The estimate of $\frac{defects}{KSLOC}$ combines all of the calculations into one.

The defects submodels use the same attributes COCOMO \tab{cocomoii-coefficients}, but also added Defect Removeal (DR) attributes.

{\small\begin{equation}\label{eq:coqualmo-defects-per-stage}
\frac{defects_{stage}}{KSLOC}=C_{stage}*DI_{stage}*DRR_{stage}
\end{equation}}

{\small\begin{equation}\label{eq:coqualmo-defects-introduced-per-stage}
DI_{stage}=
\end{equation}}

{\small\begin{equation}\label{eq:coqualmo-defects-ramaining-ratio-per-stage}
DRR_{stage}=\prod_k^3{(1-DRF_{stage,DR_k})}
\end{equation}}

TODO: finish equations

\input{tables/coqualmo-coefficient-values}

\input{tables/coqualmo-defect-stage-baselines}

COQUALMO follows the same convention as COCOMO for the effort multipliers; i.e.
nominal values ($n=3$) add nothing to the predicted number of defects. As
above, COQUALMO is:
\begin{equation}\label{eq:coqualmo-di-linear-tuning-model}\small
\begin{array}{l}
\forall x \in \{1..6\}\; ATTR_c=m_c(x-3) + 1
\end{array}
\end{equation}
where $ATTR$ is any effort multipler or scale factor from the COCOMO\index{COCOCMO} model.  
$m_c$ denotes the effect of $c$ on \textit{defect introduction}.

Simlar to the effort model\subsec{subsec:xomo-effort}, the effort multipliers and scale factors form two groups:
\begin{enumerate}
\item The \textit{positive defect} features, slopes $m^+_c$, 
are proportional to the estimated number of introduced defects:
flex,
data,
ruse,
cplx,
time,
stor, and
pvol.
\item The \textit{negative defect} features, slopes $m^-_c$, 
are inversely proportional to the estimated number of introduced defects:
acap,
pcap,
pcon,
apex,
plex,
ltex,
tool,
site,
sced,
prec,
resl,
team,
pmat,
rely, and
docu.
\end{enumerate}
The range of tunings for defect introducing features are
\begin{equation}\label{eq:di-linear-tuning-model-slope-range}\small
\begin{array}{c}
requirements\left\{
\begin{array}{c}
     0 \le m^+_c \le  0.112\\
-0.183 \le m^-_c \le -0.035
\end{array}
\right. \\
~\\
design \left\{
\begin{array}{c}
     0 \le m^+_c \le  0.14\\
-0.208 \le m^-_c \le -0.048
\end{array}
\right. \\
~\\
coding \left\{
\begin{array}{c}
    0 \le m^+_c \le  0.140\\
-0.19 \le m^-_c \le -0.053
\end{array}
\right.
\end{array}
\end{equation}

\noindent
The range of tunings for defect removal features are:
\begin{equation}\label{eq:drf-linear-tuning-model-slope-range}\small
\begin{array}{rl}
\forall x \in \{1..6\} &DR_i=m_d(x - 1)\\
requirements: 	& 0.08 	\le m_d \le 0.14 \\
design: 	& 0.1 	\le m_d \le 0.156 \\
coding: 	& 0.11 	\le m_d \le 0.176 \\
\end{array}
\end{equation}
where $m_d$ denotes the effect of $d$ on \textit{defect removal}.

\subsection{Threat \cite{madachy94}}\label{sec:threat}

The THREAT model
returns a heuristic estimate of the threat of a
schedule over run in the project.
This estimation model is dependent upon
the COCOMO effort multipliers

Internally, THREAT contains dozens of tables of the
form of \ref{fig:madeg}. Each such table  adds some ``threat'' value
to the overall project risk when multiplied by
the effort multiplier values of the corresponding COCOMO attributes.
 There are six major categories: schedule, product, personnel, process, platform and reuse.
After the threat for each category is calculated, the sum is
normalized to produce the final threat rating.

\ref{tab:threat-example-table} can be represented as
an exponentially decaying function that peaks in one corner of the risk table
at a value of two.
All the tables peak at either a value of two  or four.
Since this model is heuristic in nature, the exact height of the peak is
not certain.
When we perform Monte Carlo simulations over THREAT, we vary the height of
the peak by
a random factor $0.5 \le x \le 1$ if the peak is four, and $0.5 \le x \le 1.5$ if the peak is two.

\input{calculations/madachy-threat-total-threat-calculations}

\input{tables/madachy-threat-lookup-tables}


\section{Verification}

Before any experiments can be trusted, the \xomo{} model used in \nova{} needs to be verified.

\subsection{Verification of \xomo{} Model scoring}
\subsubsection{Overview}
Two external ``Oracles'' were used to verify NOVA's XOMO model scoring:
Ray Madachy's COCOMO II with Heuristic Risk Assessment \cite{expert-cocomoii} website for COCOMO II effort, months, and threats and
a NASA spreadsheet \cite{nasa-spreadsheet} for COQUALMO defects.

\subsubsection{Process}
Ray Madachy's website \cite{expert-cocomoii} was simple to interact with given it is a website.
To score our generated projects, python code \code{cocomo2000web.py} was written to send the project information to the website and parse the results.

On the other hand, the NASA spreadsheet \cite{nasa-spreadsheet} was difficult to interact with for automated verification.
Therefore it was coded up in python \code{coqualmo.py}.

Finally, a small python app \code{verify-xomo.py} was written to take projects generated by \nova{},
score them using \code{cocomo2000web.py} and \code{coqualmo.py},
and compare \nova{} scores verses scores from the ``Oracles''.


\subsubsection{Results}
The graphs in \fig{verification-xomo-model} show \ac{MRE}\footnote{$MRE=\frac{|actual - guess|}{actual}$}
with the scores from \nova{} as the guesses, and the scores from the ``Oracles'' as the actuals.
Each line per graph represents a different \ac{NASA} case study (OSP, OSP2, ground, flight, ALL).
For each case study, 1000 projects were generated and scored by both \nova{} and the ``Oracles''.
To make the graph easier to see, the scores were the sorted.

Ideally, the lines on the graphs will be on the X-axis.  This choice of graph was chosen to mirror
graphs\cite{star-paper-with-verification} generated by STAR.  The larger \ac{MRE} values are there because \nova{}
doesn't assume any local calibration, so the tuning parameters of the models in \xomo{} are free to vary within their allowable ranges.

\input{figures/verification-xomo-model}

\subsection{LC-COMPARO}

\section{Confirming Previous Experiments} %TODO: better seciton title
\subsection{BFC}
\subsection{Hohin}

\section{New Experiments} %TODO: better seciton title
\subsection{Compare SEESAW-[RD] vs other search engines} %TODO: better section title
\subsection{Comparing Value based evaluation vs energy} %TODO: better section title

\section{XML Interface}
\subsection{NOVA-WEB}
\lstinputlisting[language=xml,title=example/example-learning-input.xml,caption=Example XML Learning Input]{code/example/example-learning-input.xml}
\lstinputlisting[language=xml,title=example/example-learning-output.xml,caption=Example XML Learning Output]{code/example/example-learning-output.xml}
\lstinputlisting[language=xml,title=example/example-error-output.xml,caption=Example XML Error]{code/example/example-error-output.xml}
