\documentclass{article}
\usepackage{url}
\newcommand{\bi}{\begin{itemize}}
\newcommand{\ei}{\end{itemize}}
\newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\tion}[1]{\S\ref{tion:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{itemize}
   \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{itemize}}
\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}


\begin{document}
More importantly than the above are the discussions we have about the models. The following section
is not a formal analysis. Rather, it summarizes the common dialogues we have
when we present the above results.  

One reaction we get is {\em over-acceptance}. We have found one audience that are uncritical of our work
and want to implement our results, immediately. Typically, this audience is formed of managers
who are nervous of their current practices and aver (overly) eager to change to any other approach.

While this is a gratifying response, we have found that
members of this audience may  not fully appreciate a important part of the above conclusions; namely that 
any policy that looks useful in general may not be useful in particular cases. Before implementing
any of the above, we strongly recommend that users use our tools to find specialized policies that do
best for their particular site.

Another reaction we get is our conclusions come from untuned COCOMO models and {\em the above policies only
make sense after local tuning}. In reply, we offer separate comments on the mean and variance of the estimates
generated by our approach.

{\em Issues with Mean:} The $experiment=1$ results of \fig{delta} shows the differences between the estimates
generated {\em with} and {\em without} tuning on local data (the no-tuning estimates came from our search-based tool and the
tuning-results can from
on-line versions of the standard USC models\footnote{COCOMO II with Heuristic Risk Assessment: \\http://sunset.usc.edu/research/COCOMOII/expert\_cocomo/expert\_cocomo2000.html}).
The $experiment=2$ results of that figure show
the mean errors seen 
when estimates are generated using conventional methods. Note that the mean differences in the $experiment=1$
results are small (36\% or less).
Further,
the mean effort values of the conventional methods is larger (never less than 42\%).
That is, \fig{delta} shows no evidence that the differences between our methods and conventional methods
is inordinately large.

{\em Issues with Variance:}
Our tools sample across the space of possible tunings. The variance of the conclusions reached in that sample
could be reduced by the use of local data to pre-tune our models, before we perform any sampling.
However, it must be noted that the variances of our conclusions are quite small (see \fig{r1} and \fig{delta}).
In summary, while we believe it may be useful to tune with local data, the overall benefit on reducing the variance or the mean  not be large.


\begin{figure}
\small
\begin{tabular}{r|rrrrr}
        &            & \multicolumn{4}{c}{data set}\\\cline{3-6}
        & experiment & flight        & ground & OSP         & OSP2        \\
\hline
defects & 1          & 30 $\pm$ 23   & 28 $\pm$ 33 & 39 $\pm$ 33 & 32 $\pm$ 18 \\
 months & 1          & 26 $\pm$ 17   & 27 $\pm$ 16 & 23 $\pm$ 17 & 12 $\pm$ 7  \\
 effort & 1          & 36 $\pm$ 31   & 34 $\pm$ 31 & 20 $\pm$ 17 & 18 $\pm$ 13 \\
\hline
 effort & 2          & 41 $\pm$ 40   &        & 44 $\pm$ 63 &  42 $\pm$ 57
\end{tabular}
     \caption{Results (Mean $\pm$ standard deviations) from our experiments.
For an explanation of the method by which this was generated, see \fig{deletahow}.
}\caption{fig:delta}
\end{figure}
\begin{figure}
\small
\begin{center}
\begin{tabular}{|p{.95\linewidth}|}\hline
The lines labeled $experiment=1$ in \fig{delta} were generated as follows:
\be
\item
Generate 100 {\em projects} based on constraints $C$ by using our search-based tool and score them.
\item
Score {\em projects}
against the standard on-line USC
models \footnote{COCOMO II with Heuristic Risk Assessment: \\http://sunset.usc.edu/research/COCOMOII/expert\_cocomo/expert\_cocomo2000.html};
\item
Combine the {\em estimate} scores from search-based tool with the {\em baseline} scores from standard on-line USC models
using $delta= \frac{estimate - baseline}{baseline}$. 
\ee
The line labeled $experiment=2$ in \fig{delta}
was generated by taking flight systems/OSP/OSP2 systems
and estimating them using a COCOMO model tuned to some NASA COCOMO, as follows:
\bi
\item
Using the NASA93 COCOMO data set\footnote{\url{http://promisedata.org/?s=nasa93}}
we tuned a COCOMO model using
using Boehem's LC method~\cite{boehm81};
\item
Generate 100 {\em projects} based on constraints $C$ by using our search-based tool and score them.
\item Effort estimates for the generated projects were generated using the tuned
COCOMO model.  
\item The means and standard deviations for those estimates were recorded in \fig{delta}.
\ei\\\hline
\end{tabular}
\end{center}
\caption{Generation method for the \fig{exp2} results.}\label{fig:deltahow}
\end{figure}
\end{document}
