\documentclass[times, twocolumn]{article} 
\usepackage{cite,paper}
\usepackage{times}
\usepackage{graphicx}
\usepackage{url}
\usepackage{alltt}
%\usepackage{quartileChart}
\newenvironment{smallitem}{
   \setlength{\topsep}{0pt}
   \setlength{\partopsep}{0pt}
   \setlength{\parskip}{0pt}
   \begin{itemize}
   \setlength{\leftmargin}{.2in}
   \setlength{\parsep}{0pt}
   \setlength{\parskip}{0pt}
   \setlength{\itemsep}{0pt}}{\end{itemize}}

\newenvironment{smallnums}{
   \setlength{\topsep}{0pt}
   \setlength{\partopsep}{0pt}
   \setlength{\parskip}{0pt}
   \begin{enumerate}
   \setlength{\leftmargin}{.2in}
   \setlength{\parsep}{0pt}
   \setlength{\parskip}{0pt} \setlength{\itemsep}{0pt}}{\end{enumerate}}

\newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
\newcommand{\be}{\begin{smallnums}}
\newcommand{\ee}{\end{smallnums}}
%\documentstyle[times,art10,twocolumn,latex8]{article}

%------------------------------------------------------------------------- 
% take the % away on next line to produce the final camera-ready version 
\thispagestyle{empty}

%------------------------------------------------------------------------- 
\begin{document}

\title{Column Pruning Beats Stratification in Effort Estimation}
%Row Pruning (a.k.a. Stratification) Does Not Improve Effort Estimation}
%Simpler, Model-Based, Software Effort Estimation with ${\mu}COCOMO$
%}

\author{
Omid Jalali, Tim Menzies, Dan Baker\\
Lane Dept. CS \& EE \\
West Virginia University, USA\\
\small{\url{ojalali@mix.wvu.edu}, \url{tim@menzies.us}}, \small{\url{dbaker6@mix.wvu.edu}}
\and
Jairus Hihn
\thanks{
The research described in this paper was carried out at West Virginia University and
the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the
National Aeronautics and Space Administration, funded by the 
NASA Software Assurance Research Program. Reference herein to any specific 
commercial product, process, or service by trade name, trademark, manufacturer, 
or otherwise, does not constitute or imply its endorsement by the United States Government.}
\thanks{
See \protect\url{http://menzies.us/pdf/07micrococomo.pdf} for an earlier draft of this paper.}\\
Jet Propulsion Laboratory,\\
Pasadena, California, USA\\
\small{\url{jairus.m.hihn@jpl.nasa.gov}}
}

\maketitle
\thispagestyle{empty}

%------------------------------------------------------------------------- 

\begin{abstract}

Local calibration combined with stratification, also known as row pruning, is a common 
technique used by cost estimation professionals to improve model performance. The results 
presented in this paper raise several serious questions concerning the benefits of row 
pruning for improving effort estimation indicating the need to rethink standard practice. 
Firstly, the mean size of improvements from row pruning appears to be relatively small 
compared to the size of the standard deviations in effort estimation data.
Secondly, the advantages of row pruning especially for the purposes of deleting spurious 
outliers can be achieved using column pruning much more effectively. Hence, we advise 
against row pruning and advocate column pruning instead.

%Prior results about the benefits of row pruning 
%for improving effort estimation must be questioned. 
%Firstly, the mean size of those improvements are quite small
%compared to the size of the standard deviations in effort estimation.
%Secondly,
%the
%advantages of row pruning (deleting of spurious outliers) can be
%achieved using column pruning.  
%In the case where  only
%some of the variables are problematic, column pruning can delete {\em
%part} of a row while preserving the rest of the information in a
%row.
%Hence, we advise against row pruning and advocate  column pruning instead.
%In the context of COCOMO-based parametric effort estimation, the following methods
%were found to be inferior to other methods: (1)~nearest neighbor stratification, (2)~model trees, 
%and (3)~linear regression. Other methods such as (4)~complex feature subset selection; and
%(5)~reuse of precise regression values from old projects were found to offer no advantage over simpler alternatives.
\end{abstract}

%------------------------------------------------------------------------- 

\Section{Introduction}
Many believe that software effort models are a waste of time.
Jorgensen reports that the majority of software effort estimates 
are expert-based and are not automatically generated from models~\cite{jorg04}. 
Effort estimation requires generalizing from a small number of old projects. 
Generalization from such limited experience is an inherently under-constrained 
problem. Hence, the learned effort models can exhibit large deviations.
For example, we reported at PROMISE 2006 that the standard deviation
on the mean {\em magnitude of the relative error}, or MRE\footnote{MRE = $abs(predicted - actual)/actual$.},
of effort estimations in 28 data sets was enormous:
\[{
\scriptsize
\{min\%,median\%,max\%\} = \{45\%, 157\%, 649\%\}
}\]
Such large deviations confuse both the generated estimates and attempts
to compare one effort estimation method with another. For example, 
Chulani \& Boehm~\cite{chulani99} and Shepperd \& Schofield~\cite{shepperd97}
report that {\em row pruning}, also known as {\em stratification}, of training data 
into related subsets improved estimation PRED\footnote{PRED(N) is the percent 
of test cases with an MRE less than N\%.} by 5\% to 12\% and 4\% to 44\% (respectively).
When compared to standard deviations in the range 45\% to 649\%, the
estimation deltas reported by Chulani \& Boehm and Shepperd \& Schofield may 
be statistically insignificant.

Regardless of their drawbacks, various institutions demand that managers produce 
model-based estimates:

\bi
\item When lobbying for resources, or arguing against funding cuts, model-based 
methods are useful tools to develop, certify, and defend an estimate. 
\item Models not only let us make decisions; they let us audit them as well.
Numerous American government bodies (DoD, NASA) require audit-able effort estimates.  
\item Models let us ``look before we leap''; e.g. effort models allow a manager to
explore and assess numerous alternate scenarios in the planning stages of a project.
\item Models can help train newcomers who can browse models to learn the processes of 
an organization.
\ei

Since model-based estimates are so useful, and since the deviation problem undermines 
the authority of those estimates, we have been exploring methods for reducing 
the deviation problem. Miller shows theoretically that decreasing the number of 
variables decreases the deviation of a linear model learned from those variables by 
minimizing least squares error~\cite{miller02}. That is, the fewer the variables, 
the more restrained are the model predictions. Inspired by Miller, and certain 
empirical results~\cite{hall03,kirsopp02}, we developed the COSEEKMO workbench~\cite{me06d}.
This COCOMO-based tool~\cite{boehm81,boehm00b} combines variable pruning with numerous alternate
model generation models (local calibration, linear regression, model trees). COSEEKMO 
assessed these alternatives via their mean performance as well as several other modeling 
criteria (reduced deviation, increased correlation, increased PRED(30)).
In the results consistent with Miller's theoretical prediction, the standard deviation 
on the MRE\% in 28 data sets fell dramatically:
\[{
\scriptsize
\{min, median, max\} : 
\overbrace{\{45, 157, 649\}}^{LC} \Longrightarrow
\overbrace{\{20, 39, 100\}}^{COSEEKMO}
}\]
(LC is the {\em local calibration} method that is standard practice in the COCOMO 
community.)

Since the deviation was reduced, it was possible to compare different methods and
select better alternatives. Hence, the mean MRE\% also decreased:
\[{
\scriptsize
\{min, median, max\} : 
\overbrace{\{43, 58, 188\}}^{LC} \Longrightarrow
\overbrace{\{22, 38, 64\}}^{COSEEKMO}
}\]
In summary, COSEEKMO implements an elaborate comparison between numerous methods. Thirty times,
the order of data in a training set is randomized, ten test cases are removed, and numerous 
techniques are applied to the remaining cases to build an effort estimator. This estimator is 
then applied to the ten test cases and comparative statistics are collected for each method.
Model selection heuristics are then applied to return the preferred model (see Figure~\ref{RejectionRules}). 
A completing justification of the design decisions made by COSEEKMO is beyond the scope 
of this paper. For full details, see~\cite{me06d}.

COSEEKMO is slow to run and after much experimentation, we have learned that most of 
that runtime is wasted. For example, COSEEKMO's model trees~\cite{quinlan92b} use 35\% of the 
CPU time yet rarely return the preferred model. Also, while certain methods are 
clearly inferior (e.g. LC), most of the remaining methods all have very similar 
performance.

Therefore, we are currently designing ${\mu}COCOMO$ (micro-COCOMO) - the subset of 
methods within full COSEEKMO that are the simplest to build, perform better than LC,
perform as well as full COSEEKMO, dramatically reduce the large deviations,
scale to larger problems, and which can be used early in the life cycle of an effort model.

An interesting result of COSEEKMO is that it strongly endorses Boehm's 1981 
research into COCOMO. Over the last three years, (starting with~\cite{me04h}), 
we have tried to improve COCOMO using modern data mining tools. To date, 
we have found no extension or variation that performs significantly better.   

There is only one area where we offer a clarification to Boehm's work. 
Prior results about the benefits of row pruning can no longer be uncritically 
accepted. For example, Chulani \& Boehm~\cite{chulani99}, report that row 
pruning (stratification) plus local calibration improved their mean PRED 
values by 5\% to 12\%. Given the large deviations seen in the top sd(MRE) plot of 
Figure~\ref{Neighborhoods}, such small improvements may be due to simple noise. 
We will show that based on the $NASA93$ data set, the advantages of row 
pruning (deleting of spurious outliers) can be achieved using column pruning. 
Further, in the case where only some of the variables are noisy, column pruning 
can delete part of a row while preserving the rest of the information in a row.

This report discusses experiments that assess the merits of adding pruning operators
into ${\mu}COCOMO$. The original version of COSEEKMO just pruned columns (variables).
An alternate strategy is to augment or replace column pruning with row pruning. 
The motivation for row pruning, also known as {\em stratification}, is very 
clear. In a COCOMO data set, each row is one project and data collected in 
one context (e.g. NASA flight software) may not be completely relevant to 
some other context. In theory, similar projects have less variation and so 
can be easier to calibrate. Nevertheless, we show here that in many 
situations, row pruning offers no advantage over other methods.
Hence, ${\mu}COCOMO$ will perform column pruning, but not row pruning.

%------------------------------------------------------------------------- 

%\Section{Why Explore ${\mu}COCOMO$?}
%
%In some sense, ${\mu}COCOMO$ is no better than full COSEEKMO. Both systems are 
%implemented. Both are released under an open source license and can be freely 
%downloaded\footnote{From \url{http://unbox.org/wisp/tags/coseekmo}}. 
%Both find models of similar value.
%
%Nevertheless, there are many reasons to explore ${\mu}COCOMO$. ${\mu}COCOMO$ directly 
%addresses the goals of the PROMISE workshop. PROMISE aims at encouraging experiments 
%that can be reproduced, improved, and maybe even refuted. One way to facilitate this is 
%to simplify the construction of the experimental rig. Implementing full COSEEKMO was 
%an exacting task and it would take other researchers weeks to months to reproduce it. 
%${\mu}COCOMO$ is a (very) simple system that can be used in place of full COSEEKMO 
%for most applications. Also, ${\mu}COCOMO$ would be very easy to (re)implement, extend, and modify.
%
%Furthermore, ${\mu}COCOMO$ forces an audit of effort modeling methods. Such an audit 
%is long overdue. Many effort estimation methods such as clustering~\cite{garre05}, 
%neural networks~\cite{mair00investigation}, and case-based reasoning~\cite{shepperd97} 
%exist in the literature. In industry, the most commonly-used formal techniques are 
%parametric or regression-based techniques, as used in 
%COCOMO-I~\cite{boehm81}, COCOMO-II~\cite{boehm00b}, SLIM~\cite{putnam92}, SEER-SEM~\cite{jensen83}, and PRICE-S~\cite{park88}.
%%\mbox{COCOMO-I~\cite{boehm81}}, %\mbox{COCOMO-II~\cite{boehm00b}}, %SLIM~\cite{putnam92}, %\mbox{SEER-SEM~\cite{jensen83}}, and
%%\mbox{PRICE-S~\cite{park88}}.
%There are very few published studies that empirically compare this diverse set. 
%What is more usual are narrowly focused studies, such as those conducted by 
%Kemerer~\cite{kemerer87}, Briand~\cite{briand00}, Lum~\cite{lum02}, or 
%Ferens~\cite{ferens98} that test linear regression models in different environments.
%
%The reason for this lack of empirical studies is two-fold. As mentioned in the 
%introduction, most effort estimation is expert-based, not model-based, and 
%expert-based evaluations are difficult to repeat. Also, the large deviation problem 
%makes it very difficult to come to definitive conclusions about the relative
%merits of different model-based methods. 
%
%COSEEKMO's ability to reduce that 
%variance should changes the nature of empirical effort estimation research. 
%It is now possible to compare, and reject, certain model-based methods as (i)~inferior 
%or (ii)~competitive in terms of MRE or PRED, but inferior in terms of implementation
%complexity or runtime.
%We therefore anticipate a few years of COSEEKMO/${\mu}COCOMO$-style experimentation 
%where model-based methods are debated before some consensus emerges on best-practices 
%for model-based effort estimation. Once that consensus is available, the field 
%can move on to comparing our best model-based methods with expert-based estimation.
%

%In 1981, Boehm defined the COCOMO-I parametric software effort estimation tool~\cite{boehm81}.
%COCOMO assumed that effort was exponentially proportional to
%the size of the software and linearly proportional to a set of effort multipliers $EM_i$ such as 
%analyst capability
%\begin{equation}\label{eq:one}
%effort = a* SLOC ^b*\left(\prod_i EM_i\right) 
%\end{equation}
%where $a,b$ were calibration parameters that let analysts tune their model to the local domain.
%As shown in Figure~\ref{LocalCalibrationProcedure}, COCOMO-I {\em local calibration} method was a simple linear regression that assumed constant
%$EM_i$ values and just altered $a,b$.
%Boehm validated COCOMO-I against $coc81$- a data set of 63 projects (available in the
%PROMISE
%repository).
%
%(Subsequent work by Boehm offered
%some modifications to the effort multipliers and added some scale
%factors into $b$~\cite{boehm00b}. This paper does not explore those modifications since the
%COCOMO-II validation data set is confidential and not available in PROMISE.).
%
%This paper audits Equation~\ref{eq:one} using $nasa93$ (some new COCOMO-I style data available in the
%PROMISE repository) as well as 16 subsets of $coc81$ and $nasa93$ (each subset contains data
%stratified to a particular source such as just ground systems, or just those written in FORTRAN, etc).
%We will endorse the core intuitions of that equation, but reject certain superfluous over-elaborations.
%For example:
%\begin{itemize}
%\item Not all the COCOMO variables  are required for all effort estimation problems.
%Given a particular project to estimate, 
%pruning away certain COCOMO variables decreases mean error (a little)
%and the variance on the error (a lot).
%\item
%After pruning, and before local calibration, Boehm's effort multiplets
%can be replaced with three sets
%of {\em proximal values} (defined
%to only two significant figures) without increasing estimation error. 
%\item
%A simple linear-time pruning method works as well as far more
%complex method (such as those we have explored previously~\cite{me06d,me05c,me05a}.
%\item
%Numerous other schemes for generating effort estimates (model trees, linear regression,
%on all the COCOMO parameters, nearest neighbor methods) 
%do no better than local calibration on pruned sets.
%\end{itemize}
%These results are quite significant:
%\begin{itemize}
%\item
%The proximal results mean
%COCOMO, with variable
%pruning, can be deployed into a new organization without time consuming recalculation of the
%effort multipliers.
%\item
%The linear-time pruning
%results mean that very rapid effort estimation
%and we will show
%that proximal values while detailed numeric values for the effort mul
%\end{itemize}
%
%Local calibration (LC) \cite{boehm81} is used widely as the standard 
%practice in cost estimation. However, the results obtained using LC 
%have large deviations \cite{me06d}. Previously, COSEEKMO was designed to 
%explore different model-based methods in cost estimation aiming to reduce such 
%large deviations. Prior to this paper, it was shown that methods used 
%in COSEEKMO reduce the deviations \cite{me06d}. This was done by trying 
%all combinations of proximal and precise estimations for numeric values, 
%feature subset selection methods, and different learners such as M5P, LSR, 
%and standard LC. A new method for reducing the deviations is studied in 
%this paper. This method only uses proximal numeric estimations followed by 
%a simpler feature subset selection method (column pruning) and finally local 
%calibration as the learner. The new method gives deviations better than the 
%standard practice and no worse than what was reported from COSEEKMO. Since 
%COSEEKMO tries all combinations and reports the best, and since the new 
%method gives no worse deviations, and usually much better, it is clear that 
%the new method is an improvement over the current methods in cost estimation 
%aiming to reduce the deviation.
%
%------------------------------------------------------------------------- 

\Section{Background}

COSEEKMO is based around COCOMO. The core intuition behind COCOMO is as a program 
grows in size, the development effort grows exponentially. More specifically:

\begin{displaymath}
%\label{eq:one}
{\footnotesize effort(person months)=a*\left(KLOC^b\right)*\left(\prod_jEM_j\right)}
\end{displaymath}

Here, $KLOC$ is thousands of delivered source instructions, $EM_j$ is one of 
Figure~\ref{EffortMultipliers} effort multipliers with precise values from 
Figure~\ref{PreciseEffortMultipliers}, and $a$ and $b$ are the parameters 
tuned by Boehm's local calibration (LC) procedure of Figure~\ref{LocalCalibrationProcedure}.

\begin{figure}[!t]
{\footnotesize\begin{alltt}\sffamily
\textbf{FUNCTION} worse(x,y)
    \textbf{IF} statisticallyDifferent(x,y) \textbf{THEN}
        \textbf{IF} error(x) \textless error(y)                    \textbf{THEN} \textbf{RETURN} y \textbf{FI} {\em rule1}
        \textbf{IF} error(y) \textless error(x)                    \textbf{THEN} \textbf{RETURN} x \textbf{FI} {\em rule1}
    \textbf{ELSE}    
        \textbf{IF} sd(x) \textless sd(y)                            \textbf{THEN} \textbf{RETURN} y \textbf{FI} {\em rule2}
        \textbf{IF} sd(y) \textless sd(x)                            \textbf{THEN} \textbf{RETURN} x \textbf{FI} {\em rule2}

        \textbf{IF} correlation(x) \textless correlation(y)  \textbf{THEN} \textbf{RETURN} x \textbf{FI} {\em rule3}
        \textbf{IF} correlation(y) \textless correlation(x)  \textbf{THEN} \textbf{RETURN} y \textbf{FI} {\em rule3}

        \textbf{IF} pred(x) \textless pred(y)                     \textbf{THEN} \textbf{RETURN} x \textbf{FI} {\em rule4}
        \textbf{IF} pred(y) \textless pred(x)                     \textbf{THEN} \textbf{RETURN} y \textbf{FI} {\em rule4}

        \textbf{IF} \(|Subset(x)|\) \textless \(|Subset(y)|\)       \textbf{THEN} \textbf{RETURN} y \textbf{FI} {\em rule5}
        \textbf{IF} \(|Subset(y)|\) \textless \(|Subset(x)|\)       \textbf{THEN} \textbf{RETURN} x \textbf{FI} {\em rule5}
    \textbf{FI}
\textbf{RETURN} 0 {\em if no reason to return true}
\end{alltt}}
\caption{COSEEKMO rejection rules (slightly updated from~\cite{me06d}). 
{\em Error} refers to MMRE. {\em Worse}'s statistically difference test 
compares two MMREs $x$ and $y$ using a two-tailed t-test at the 95\% 
confidence interval.
}
\label{RejectionRules}
\end{figure}

\begin{figure}[!t]
\begin{center}
{\scriptsize
\begin{tabular}{l|r@{:~}l|}\cline{2-3}
upper:   &ACAP&analysts capability\\
increase &PCAP&programmers capability\\
these to &AEXP&application experience\\
decrease &MODP&modern programming practices\\
effort   &TOOL&use of software tools\\
         &VEXP&virtual machine experience\\
         &LEXP&language experience\\\hline
middle   &SCED&schedule constraint\\\hline
lower:   &DATA&data base size\\
decrease &TURN&turnaround time\\
these to &VIRT&machine volatility\\
increase &STOR&main memory constraint\\
effort   &TIME&time constraint for cpu\\
         &RELY&required software reliability\\
         &CPLX&process complexity\\\cline{2-3}
\end{tabular}}
\end{center}
\caption{COCOMO 81 effort multipliers.}
\label{EffortMultipliers}
\end{figure}

\begin{figure}[!t]
\begin{center}
{\scriptsize
\begin{tabular}{|l|c|r@{~}|r@{~}|r@{~}|r@{~}|r@{~}|r|}
    \hline
&    &very&&&&very&extra\\
    &&low&low&nominal&high&high&high\\
    \hline
upper&ACAP   &1.46   &1.19   &1.00   &0.86   &0.71   &\\
(increase&PCAP   &1.42 &1.17   &1.00   &0.86   &0.70 &\\
these to&AEXP   &1.29 &1.13   &1.00   &0.91   &0.82   &\\
decrease&MODP   &1.2  &1.10 &1.00 &0.91 &0.82 &\\
effort)&TOOL   &1.24 &1.10 &1.00 &0.91 &0.83 &\\
&VEXP   &1.21 &1.10 &1.00 &0.90 &  &\\
&LEXP   &1.14 &1.07 &1.00 &0.95 &  &\\\hline
middle&SCED   &1.23 &1.08 &1.00 &1.04 &1.10 &  \\\hline
lower&DATA   &    & 0.94 &1.00 &1.08 &1.16&\\
(increase&TURN   &       &0.87   &1.00   &1.07   &1.15   &\\
these to&VIRT   &       &0.87   &1.00   &1.15   &1.30   &\\
increase&STOR   &       &       &1.00   &1.06   &1.21   &1.56\\
effort)&TIME   &  &    &1.00   &1.11   &1.30   &1.66\\
&RELY   &0.75& 0.88& 1.00 & 1.15 & 1.40&\\
&CPLX   &0.70 &0.85 &1.00 &1.15 &1.30 &1.65\\
    \hline
\end{tabular}}
\end{center}
\caption{Precise values for Figure~\protect\ref{EffortMultipliers}.}
\label{PreciseEffortMultipliers}
\end{figure}

\begin{figure}[!t]
\footnotesize
\begin{center}
\begin{tabular}{|p{3in}|}\hline
The matrix $D_{i,j}$ holds
\bi
\item the natural logs of the $KLOC$ estimates,
\item the natural logs of the actual efforts for projects $i \le~$j$~\le t$,
\item the natural logs of the cost drivers (the scale factors and effort 
multipliers) at locations $1 \le i \le 15$ (for COCOMO 81) or $1 \le i \le 22$ 
(for COCOMO-II).
\ei
The following calculation yields estimates for ``$a$'' and ``$b$'' that minimizes the sum of 
the squares of residual errors between predicted and actual efforts in a COCOMO model.

\begin{displaymath}
\begin{array}{rcl}
EAF_i& =& \sum_j^N D_{i,j}\\
a_0  & =& t \\
a_1  & =& \sum_i^t KLOC_i\\
a_2  & =& \sum_i^t (KLOC_i)^2\\
d_0  & =& \sum_i^t \left(actual_i - EAF_i\right)\\
d_1  & =& \sum_i^t \left((actual_i - EAF_i)*KLOC_i\right)\\
b  & =& (a_0d_1 - a_1*d_0)/(a_0a_2 - a_1^2)\\
a_3    & =& (a_2d_0 - a_1d_1)/(a_0a_2 - a_1^2)\\
a    & =& e^{a_3}
\end{array}
\end{displaymath}\\\hline
\end{tabular}
\end{center}
\caption{LC: local calibration (from ~\cite{boehm81}).}
\label{LocalCalibrationProcedure}
\end{figure}

We use COCOMO since, unlike other models such as PRICE-S~\cite{park88}, 
SLIM~\cite{putnam92}, or SEER-SEM~\cite{jensen83}, it is an open 
model with published data. All details were published in Boehm's 
text {\em Software Engineering Economics}~\cite{boehm81,boehm00b}.
There are two version of COCOMO: COCOMO-I and COCOMO-II. 
We base our studies on COCOMO-I data since we know of no large public 
domain COCOMO-II data sets. In contrast, several COCOMO-I data sets 
are available in the PROMISE repository~\cite{me07c}:

\bi
\item $COC81$: The 63 projects used to define COCOMO-I. This data comes 
from a variety of domains including engineering, science, financial, etc. 
\item $NASA93$: The 93 NASA projects from the 1980s and 1990s. Since this 
data comes from NASA, it is stratified to just aerospace applications.
\ei

%------------------------------------------------------------------------- 

\Section{External Validity}

The external validity of our study must be discussed. These results are only
relevant to COCOMO-style parametric effort estimation. They are based only 
on the projects described in the $COC81$ and $NASA93$ data sets. Nevertheless, 
to the best of our knowledge, $COC81$ and $NASA93$ represent the universe of 
publicly available COCOMO data sets.

Also, our results are limited to the space of learners, WRAPPERS, and other 
methods explored by COSEEKMO and ${\mu}COCOMO$, which are:

\bi
\item local calibration,
\item model trees,
\item linear regression,
\item LocalWRAPPER, and
\item Euclidean nearest neighbor.
\ei

It is possible that other methods would outperform and overturn our results. 
In the true spirit of PROMISE, all our data and tools are on-line. We invite 
other researchers to download them, extend them, and to supersede our results.

%------------------------------------------------------------------------- 

\Section{Experiments}

%------------------------------------------------------------------------- 

\subsection{Prior Row Pruning Experiments}

Our previous experiments were quite negative on the merits of row pruning.
As reported in \cite{me06d}, the rows of $NASA93$ divide into various subsets:

\bi
\item All the records, 
\item Whether or not it is flight or ground system, 
\item The parent project that includes the project expressed in this record, 
\item The NASA center where the work was conducted, 
\item etc. 
\ei

Some of these subsets are bigger than others. Given a record labeled with subsets 
$\{S_1, S_2, S_3 , . . .\}$ then we say a {\em candidate stratification} has 
several properties: 
\bi
\item It contains records labeled $\{S_i, S_j\}$. 
\item The number of records in $S_j$ is greater than $S_i$; i.e. $S_j$ 
is the superset and $S_i$ is the stratification. 
\item $S_j$ contains 150\% (or more) of records in $S_i$. 
\item $S_i$ contains at least 20 records. 
\ei

COSEEKMO searched for effort models that were better in the subset than the superset.
$NASA93$ contains 207 candidate stratifications (including one stratification 
containing all the records), and better effort models were found in only four cases.
That is, in the overwhelming majority of cases, row pruning (stratification) offered
no advantage.

%------------------------------------------------------------------------- 

\subsection{New Row Pruning Experiments}

Another possibility, not previously explored, is that the stratifications are mis-labeled.
The stratifications seen in $COC81$ and $NASA93$ come from row labels. If domain experts 
do not fully understand the factors that affect software costs, they might inadvertently group 
together the wrong rows.

To test that possibility, we performed row pruning experiments where stratifications 
were selected from the training set via their Euclidean distance to the test case. 
That is, for each member of the test set, an effort model is generated (using LC) 
from the closest $N$ members of the training set. For this experiment, $distance$ was 
defined using:
\[{
\scriptsize
\mbox{distance case1 to case2} = \sqrt{\sum_{X \in EM} \left(\frac{X_1 - X_2}{max(X) - min(X)}\right)^2}
}\]
(where the X values are the numeric values of Figure~\ref{PreciseEffortMultipliers}).

Figure \ref{PruningTactics} shows the effects of different combinations of row and column
pruning on the standard deviation of the MRE, the mean MRE, and PRED(30).
This is our standard report format so, before reviewing those findings, we review
the details of those reports:

\bi
\item The x-axis of Figure~\ref{PruningTactics} denotes which subset of $COC81$ or $NASA93$ 
was used in this experiment.  
\item Plots are always shown in sets of three. A vertical line through the three plots would
show results from the same subset.
\item The plots were generated using the COSEEKMO method described in the introduction. 
\item For $sd(MRE)$ and $mean(MRE)$, the goal is to get $lower$ lines than the other methods.
\item For $\%pred(30)$, the goal is to  get $higher$ lines than the other methods.
\ei

\begin{figure*}[!t]
\begin{center}
\includegraphics{PruningTactics-SdMRE}
\includegraphics{PruningTactics-MeanMRE}
\includegraphics{PruningTactics-Pred}
\end{center}
\caption{Effects of different pruning tactics.}
\label{PruningTactics}
\end{figure*}

\begin{figure*}[!t]
\begin{center}
\includegraphics{LocalWrapper-Precise-LOCOMO-SdMRE}
\includegraphics{LocalWrapper-Precise-LOCOMO-MeanMRE}
\includegraphics{LocalWrapper-Precise-LOCOMO-Pred}
\end{center}
\caption{Effects of different neighborhoods.}
\label{Neighborhoods}
\end{figure*}
 
\noindent
As to the particulars of Figure~\ref{PruningTactics}: 

\bi
\item These plots show results from some preprocessing of the data followed 
by local calibration (LC). In the case of Figure~\ref{PruningTactics}, those 
preprocessors were some combination of row and column pruning. 
\item In this plot, LC used the precise effort multiplier values of Figure~\ref{PreciseEffortMultipliers}.
\item The lines labeled ``Precise-None-None-LC'' show the standard effort 
estimation method as recommended by Boehm; i.e. just perform local calibration.
\item The top two lines of each plot show a comparison between row pruning 
and no row pruning when there is no column pruning present.
\item The last two lines add in column pruning. For this figure, the column pruning method
was based on Kohavi's WRAPPER~\cite{kohavi97}. The WRAPPER passes different subsets of
variables to some oracle (in our case, LC) and returns the subset which yields the best 
performance. WRAPPER is thorough but can be slow since (in the worst case), it has to 
explore all subsets of the available variables.
\item The row pruning in these results used the $N=20$ closest cases to build an effort model 
for the test cases (and the impact of changing $N$ is discussed below).
\ei

In Figure~\ref{PruningTactics}, the following effect is a strong argument that local 
calibration, by itself, needs to be augmented or replaced: 

\bi
\item The ``Precise-None-None-LC'' results in the top $sd(MRE)$ plot show devastating 
large deviations resulting from standard methods (LC) - often over 100\% and sometimes
much larger.
\ei

The following effects are strong arguments against row pruning via Euclidean distance measures:

\bi
\item The top two plots for $sd(MRE)$ and $mean(MRE)$ show that without other 
pruning methods, row pruning does no better than LC.
\item The top two plots also show that when column pruning is active, row pruning 
adds little or no advantage.
\ei

The following effects are strong arguments for column pruning:

\bi
\item The top two plots show that the column pruning always resulted in much less error and deviation.
\item In the middle plot, column pruning without row pruning generates the smallest errors.
\ei

The only argument that can be mounted from these results against column pruning is:

\bi
\item In the bottom plot, column pruning, by itself, sometimes yields the lowest PRED.
This effect is much less marked than those discussed above. So, it does not dissuade
against our thesis that there is much merit in column pruning and little merit in row pruning.
\ei

%\begin{figure}[!t]
%\begin{center}
%{
%\scriptsize
%\include{Pruning-All-SdMRE}
%\include{Pruning-All-MeanMRE}
%\include{Pruning-All-Pred}
%}
%\end{center}
%\caption{Effects (SdMRE, MeanMRE, Pred) of different pruning tactics.}
%\label{PruningTacticsQuartileCharts}
%\end{figure}

These results run counter to several other publications 
(e.g.~\cite{chulani99,shepperd97}) including our research~\cite{me04h,me05c}
that report an improvement in PRED following stratification. One possibility 
is that the above results did not select the right number of neighbors to build an 
effort estimation model.

Figure~\ref{Neighborhoods} tests that speculation. The proceeding discussion endorsed 
column pruning with the WRAPPER followed by LC. Figure~\ref{Neighborhoods} augments 
such a rig with a nearest-neighbor section of \mbox{$N \in\{5,10,20,40,80\}$}. 
No clear ``best'' nearest-neighbor is apparent. Indeed, looking at the mean(MRE) 
results, changing the size of neighborhood had very little effect on the mean error.
All in all, there is no evidence in Figure~\ref{Neighborhoods} that the above results 
are conflated by a wrong selection of nearest neighbors. 

%------------------------------------------------------------------------- 

\Section{Discussion}

\noindent
Based on the above, we assert that:

\be
\item Column pruning is particularly useful at reducing the mean and standard deviation
of the MRE seen with COCOMO-style local calibration. This effect has been reported previously~\cite{me06d}.
\item Row pruning offers no advantage for the generation of effort models using a COCOMO-style 
LC procedure. This is a new result.
\ee

%\begin{figure}[!t]
%\begin{center}
%{
%\scriptsize
%\include{Neighborhoods-All-SdMRE}
%\include{Neighborhoods-All-MeanMRE}
%\include{Neighborhoods-All-Pred}
%}
%\end{center}
%\caption{Effects (SdMRE, MeanMRE, Pred) of different neighborhoods.}
%\label{NeighborhoodsQuartileCharts}
%\end{figure}

Conclusion \#2 is counter-intuitive. Many researchers (e.g.~\cite{chulani99,shepperd97,me04h,me05c})
advise that a standard practice for modeling is to ``remove outliers''; i.e. cull rows of 
data that contain spurious signals. We speculate that prior results regarding the merits of 
stratification may have been conflated by the large deviation effect; i.e. their reported improvements 
were merely noise within a widely variant space of solutions.

Another possibility is that the ``remove outliers'' advice is subtly misguided. In the 
case where the outliers are caused by a small number of deviant variables, then if those 
variables are removed (by column pruning), the information in the rest of the row remains 
and can assist in the modeling task. In that case, the ``remove outlier'' advantage seen 
in row pruning would also be achieved with column pruning.

Our own prior results that endorsed row pruning need further discussion. 
Chen, Menzies, Port, and Boehm (hereafter, CMPB)~\cite{me05c} concluded that ``pruning always 
improved estimation effectiveness''. That conclusion was made based on the results 
in Figure~\ref{OtherPapersResults}. The y-axis of that figure shows the PRED(30) results seen
after performing 30 linear regressions on $\frac{2}{3}$-rds of the data (random selected),
then applying the learned model to the remaining $\frac{1}{3}$-rd of the data.
The top $before$ plot shows the linear regression results while the $after$ plot shows 
the effect of linear regression plus column pruning via the WRAPPER. The data sets on the 
x-axis are sorted left-to-right according to how many records they contain (see the blue plot). 
To produce those data sets, the four data sets shown on the left hand side 
($cii0,cii4,coci,na60$) were divided two ways:

\be
\item The data was divided into the project, geographical center, and task performed
by the software. This generated the three middle data sets ($pall,call,tall$).
\item The data was divided according to their most specific stratification; e.g.
a project from a particular project at a particular center working on a particular
task. This generated the eight smallest data sets shown on the right 
($c03,c01,t03,p04,t02,c02,p02,p03)$.
\ee

That is, the data sets in the middle and on the right of Figure~\ref{OtherPapersResults} 
are stratifications (row prunings) of data shown on the left. 

From those results, CMPB concluded a column pruning, plus extensive row pruning, was the best 
effort estimation policy. However, those results applied row and column pruning to non-identical 
portions of data sets when comparing different combinations of column and row pruning.

In contrast, the analysis discussed here compares row pruning separately and in combination with column 
pruning. It also uses identical data sets in all these comparisons. Hence, this paper is a more 
complete analysis of row pruning and column pruning effects.

\begin{figure}[!t]
\begin{center}
\includegraphics[width=3.2in,angle=270]{colp}\includegraphics[width=3.2in,angle=270]{lightRowColp}\includegraphics[width=3.2in,angle=270]{heavyRowColp}
\end{center}
\caption{Results from~\cite{me05c}.}
\label{OtherPapersResults}
\end{figure}

% e but 
%this effect is not particularly strong. 
%\ei
%shows that, measured in terms of PRED, row p 
%and measured in 
%Adding row 
%pruning actually made the standard deviations larger in a few instances. 
%The bottom two lines show a comparison between row pruning and no row pruning 
%when column pruning is present. As it can be seen, row pruning does not reduce 
%the standard deviations when present. Therefore, row pruning is not a helpful method. 
%(Although \cite{me05c} shows that row pruning can be useful, its analysis 
%offers results from non-identical portions of data sets when comparing 
%different combinations of column and row pruning. The analysis here 
%offers identical data sets and hence it is a more complete comparison. 
%For a complete discussion, see the Conclusion section).
%
%The column pruning method used is the same as the one used in the previous 
%COSEEKMO \cite{me06d}. Also, the standard method, LC, is used as a baseline 
%in Figure \ref{PruningTactics} which is the same as no row pruning and 
%no column pruning and only the LC learner. In this comparison, the estimation 
%method is kept the same, as precise, which is also the standard practice. The 
%column pruning returns the attributes that are the most useful in closely 
%estimating the actual cost. 
%
%The row pruning returns the instances that are the 
%most useful in closely estimating the actual cost. Row pruning is performed 
%after column pruning since there would be fewer attributes used in finding the 
%nearest neighbors to the test instance. 
%
%The row pruning method returns the {\em n} 
%nearest instances to the given instance using their distance from the given 
%instance as a measure of how close they are to that instance. There were 
%experiments for finding the best neighborhoods and surprisingly all 
%neighborhoods followed the same pattern and performed very closely to each other 
%(Figure \ref{Neighborhoods}). Therefore, different neighborhoods perform the 
%same way and having different neighborhoods is actually not helpful. Hence, a 
%neighborhood of 80 that includes all train instances, which is essentially the 
%same as not performing row pruning, performs as good as any other neighborhood. 
%This reinstates the fact that row pruning is of little help if any.
%
%All aspects of these methods are computationally inexpensive. In the worst 
%case, the row pruning method requires O(k*(n-k)) time where n is the total 
%number of instances, n-k is the number of train instances, and k is the 
%number of test instances. Since in this experiment k is a constant (=10), 
%the running time is linear in terms of n. The column pruning method requires 
%two passes, once forward and once backward, through all the attributes 
%(explained later in Figure \ref{COCOMINAlgorithm}). Therefore, for j attributes 
%(cost drivers), the running time is O(j*n). Since j is a constant (=15), the column 
%pruning method is also linear in terms of n. The learner, LC, requires one pass 
%in the training data and another pass in the testing data. Each step in these 
%passes requires constant time. Therefore, the training pass requires O(n-k) and 
%the testing pass requires O(k) time. Overall, the learner requires O(n) time. 
%Since this method is consisted of linear time methods, in the worst case, it 
%is linear in the total number of instances, hence computationally inexpensive.
%
%%------------------------------------------------------------------------- 
%\subsubsection{Discussion}
%
%Prior reports on the benefits of row pruning/stratification may have found performance
%differences due to large deviances, and not due to true improvements.
%

%------------------------------------------------------------------------- 

\Section{Conclusion}

%Model-based effort estimation is much simpler than previously thought. Some of
%Boehm's work appears to be an over-elaboration; e.g. proximal effort multipliers do as 
%well as precise values. Numerous sophisticated model generation methods do no better
%than simpler alternatives; e.g. a simple linear-time column pruner works as well as exponential-time
%pruner; nearest neighbor methods offer no improvement over that simple column pruner.
%Widely-used methods offer no advantage while wasting CPU time; e.g. model trees.
%Simple methods run faster, scale to larger problems, and can be used early in the effort
%estimation life cycle.

From our results, we can make several comments about certain prior results~\cite{me05c,me06d}.
%,boehm81}.

%------------------------------------------------------------------------- 

\subsection{Chen, Menzies, Port, \& Boehm '05~\cite{me05c}} 

The estimation improvements attributed to row pruning by Chen et.al.~\cite{me05c} were not found here.
Those results were due to a methodological error: i.e. row and column pruning was not applied 
consistently to that data. Here, after applying column and row pruning to the same data,
we found that row pruning gave no improvement over column pruning.

%------------------------------------------------------------------------- 

\subsection{Menzies, Chen, Hihn, \& Lum '06~\cite{me06d}}

What do these results say about the COSEEKMO study~\cite{me06d}? Is it a needlessly elaborate system?

We would argue for the utility of COSEEKMO. COSEEKMO is a {\em certification} environment and ${\mu}COCOMO$ 
is a {\em production} environment. COSEEKMO is a general rig where new ideas and tools can be 
rigorously and exhaustively tested. However, it is also the flame that burns away the fat. 
Using COSEEKMO, we can find lean and fast methods for effort estimation, which can simplify the 
work of analysts performing model-based estimation. Our current plan is to code all those lean and fast 
methods into ${\mu}COCOMO$.

The design process of ${\mu}COCOMO$ imposed an audit of effort modeling methods. Such an audit 
is long overdue. Many effort estimation methods such as clustering~\cite{garre05}, 
neural networks~\cite{mair00investigation}, and case-based reasoning~\cite{shepperd97} 
exist in the literature. In industry, the most commonly-used formal techniques are 
parametric or regression-based techniques, as used in COCOMO-I~\cite{boehm81}, 
COCOMO-II~\cite{boehm00b}, SLIM~\cite{putnam92}, SEER-SEM~\cite{jensen83}, and PRICE-S~\cite{park88}.
%\mbox{COCOMO-I~\cite{boehm81}}, %\mbox{COCOMO-II~\cite{boehm00b}}, %SLIM~\cite{putnam92}, %\mbox{SEER-SEM~\cite{jensen83}}, and
%\mbox{PRICE-S~\cite{park88}}.
There are very few published studies that empirically compare this diverse set. 
What is more usual are narrowly focused studies, such as those conducted by 
Kemerer~\cite{kemerer87}, Briand~\cite{briand00}, Lum~\cite{lum02}, or 
Ferens~\cite{ferens98} that test linear regression models in different environments.

The reason for this lack of empirical studies is two-fold. As mentioned in the 
introduction, most effort estimation is expert-based, not model-based, and 
expert-based evaluations are difficult to repeat. Also, the large deviation problem 
makes it very difficult to come to definitive conclusions about the relative
merits of different model-based methods. 

COSEEKMO's ability to reduce that variance should change the nature of empirical 
effort estimation research. It is now possible to compare, and reject, certain 
model-based methods as (i)~inferior or (ii)~competitive in terms of MRE or PRED, 
but inferior in terms of implementation complexity or runtime.

We therefore anticipate a few years of COSEEKMO/${\mu}COCOMO$-style experimentation 
where model-based methods are debated before some consensus emerges on best-practices 
for model-based effort estimation. Once that consensus is available, the field 
can move on to comparing our best model-based methods with expert-based estimation.


%\subsection{Boehm '81~\cite{boehm81}} 
%
%The overall conclusion of this work is a
%resounding endorsement of Boehm's 1981 research into COCOMO. Over
%the last three years, (starting with~\cite{me04h}), we have tried
%to improve COCOMO using modern data mining tools.  To date, we have
%found no extension that performs better than the procedure defined
%by Boehm 1981; i.e.  local calibration to tune $a,b$ in the expression 
%\[{\footnotesize effort(person months)=a*\left(KLOC^b\right)*\left(\prod_jEM_j\right)}
%\]
%In fact, our improvements to COCOMO only results from pruning certain
%superfluous elaborations.  For example, we have found that column
%pruning can decrease the mean and standard deviation of the error.
%However, in other work~\cite{lum06} we have shown that different training
%sets result in different prunings.  That is, there is no single
%prune we can offer to COCOMO.  Instead, we advise modelers to try
%and collect data from {\em all} the COCOMO attributes (then let
%automatic algorithms determine which variables are not important
%for local data).

%There is only one area where we offer a  clarification to Boehm's
%work.  Prior results about the benefits of row pruning can no longer
%be uncritically accepted.  For example, Chulani \& Boehm~\cite{chulani99} report
%that row pruning (stratification) plus local calibration improved
%their mean PRED values by  5\% to 12\%. Given the
%large deviations seen Figure~\ref{Neighborhoods} in  the top sd(MRE) plot,
%such small improvements may be due to simple noise.  We suspect the
%advantages of row pruning (deleting of spurious outliers) can be
%achieved using column pruning. Further, in the case where  only
%some of the variables are noisy, column pruning can delete {\em
%part} of a row while preserving the rest of the information in a
%row.

%------------------------------------------------------------------------- 

\Section{Future Work}

It is possible that stratification to particular project types would improve 
effort estimation. What we have shown in our work so far is that, to date, 
we have not found that the current COCOMO attributes can characterize those
``particular project types''. It is possible that further details from the 
target domain could be used to improve effort estimation; e.g. when
Boetticher augmented the standard effort estimation attributes with knowledge 
of user-interface complexity, the resulting models had impressively large 
PRED values~\cite{boetticher03}. 

Therefore, over summer 2007, we plan to experiment with augmented COCOMO 
data sets that include more domain-specific descriptors of projects. 
The hypothesis is that the information in these additional columns will 
improve effort estimation; i.e. the row pruning based on these extra 
columns will decrease MRE and increase PRED.

\bibliographystyle{paper}
\bibliography{refs}
\end{document}

%------------------------------------------------------------------------- 
%------------------------------------------------------------------------- 
%------------------------------------------------------------------------- 

\subsection{Simpler Column Pruning}

Column pruning is useful. Therefore we should try to explore it a
little more. In {\em Experimental Methods for Artificial Intelligence},
Cohen advises comparing the performance of a supposedly more sophisticated 
approach against a simpler ``straw man''method~\cite[p81]{cohen95}. The 
following algorithm, called COCOMIN, has no particular theoretical foundation; 
it was just the simplest ``straw man'' we could conceive. Unlike many other 
WRAPPERS, COCOMIN does not pre-order the attributes and uses the attributes 
in the {\em given order} in its column pruning.

COCOMIN performs a linear (in terms of the number of columns) number of feature subset 
evaluations using LC (Figure \ref{LocalCalibrationProcedure}). LC is also linear 
in terms of the number of rows passed to it. COCOMIN performs these operations in
two passes as shown in Figure~\ref{COCOMINAlgorithm}:

\be
\item In pass \#1, the {\em forward select search}, COCOMIN starts with the set containing only $KLOC$.
Then effort models are built using one more variable. If the newly added variable improves mean(MRE), 
it is kept. Else, it is dropped and the search moves on to the next variable in the {\em given order}. 
LC is used in evaluating the mean(MRE) of each effort model.
\item
Pass \#2 is a {\em backwards elimination search}. This runs similar to pass \#1 but the
initial model contains all the variables and variables are removed in the {\em given order}.
\ee

When COCOMIN terminates, it returns the columns from either the forward or backwards search that
yielded the lowest mean(MRE) estimate.

\begin{figure}{\footnotesize\begin{alltt}\sffamily
# Forward-Select Search
bestForwardMMRE=\(\infty\)
ForwardSubset=\{KLOC\}
\textbf{FOR} all attributes in the{\em given order} \textbf{DO}
    add the attribute to ForwardSubset
    find newMMRE of ForwardSubset using Figure \ref{LocalCalibrationProcedure}
    \textbf{IF} newMMRE \textless bestForwardMMRE \textbf{THEN}
        bestForwardMMRE=newMMRE;
    \textbf{ELSE}
        remove the attribute from ForwardSubset

# Backward-Elimination Search
bestBackwardMMRE=\(\infty\)
BackwardSubset=all attributes
\textbf{FOR} all attributes in the{\em given order} \textbf{DO}
    remove the attribute from BackwardSubset
    find newMMRE of BackwardSubset using Figure \ref{LocalCalibrationProcedure}
    \textbf{IF} newMMRE \textless bestBackwardMMRE \textbf{THEN}
        bestBackwardMMRE=newMMRE;
    \textbf{ELSE}
        add the attribute back to BackwardSubset

\textbf{IF} bestForwardMMRE \textless bestBackwardMMRE \textbf{THEN}
    \textbf{RETURN} ForwardSubset
\textbf{ELSE}
    \textbf{RETURN} BackwardSubset
\end{alltt}}
\caption{COCOMIN's algorithm}
\label{COCOMINAlgorithm}
\end{figure}

Figure \ref{WRAPPER} compares the performance of LC vs. COCOMIN vs. LocalWRAPPER 
(the COSEEKMO WRAPPER algorithm that uses LC to assess each candidate set of columns).
LocalWRAPPER and COCOMIN outperformed LC and there was no clear winner between
LocalWRAPPER and COCOMIN. Therefore, we assert: 

\be
\item COCOMIN runs in linear time and appears to perform as well as LocalWRAPPER.
\item LocalWRAPPER runs in time that is, in worst case, exponential on the number of variables.
\item ${\mu}COCOMO$ should therefore include COCOMIN and exclude LocalWRAPPER.
\ee

This conclusion implies that ${\mu}COCOMO$ would scale to larger problems 
than full COSEEKMO. Firstly, it only uses one model generation method
while full COSEEKMO uses many. Secondly, ${\mu}COCOMO$'s variable pruner 
is much faster than the one used in COSEEKMO. ${\mu}COCOMO$ is therefore 
the preferred method if effort estimation is some subroutine inside some 
more complex simulation study such as the thousands of ``what-if'' queries 
made by TARZAN~\cite{me00e}, XOMO~\cite{me05e} or simulated annealing~\cite{me03n}.

\begin{figure*}
\begin{center}
\includegraphics{Wrapper-Precise-SdMRE}
\includegraphics{Wrapper-Precise-MeanMRE}
\includegraphics{Wrapper-Precise-Pred}
\end{center}
\caption{Effects of different column pruning methods.}
\label{WRAPPER}
\end{figure*}


%\begin{figure}[!t]
%\begin{center}
%{
%\scriptsize
%\include{Wrapper-All-SdMRE}
%\include{Wrapper-All-MeanMRE}
%\include{Wrapper-All-Pred}
%}
%\end{center}
%\caption{Effects (SdMRE, MeanMRE, Pred) for different WRAPPERS.}
%\label{WRAPPERQuartileCharts}
%\end{figure}

%------------------------------------------------------------------------- 

\subsection{Precise vs. Proximal Effort Multipliers}
The proceeding work offers a partial design for ${\mu}COCOMO$: prior to
running LC, prune the columns using COCOMIN. This section explores the 
effects of replacing the precise effort multipliers with {\em proximal} values.

Standard COCOMO uses the {\em precise} effort mulitpler values shown in 
Figure~\ref{PreciseEffortMultipliers}. These values were generated by Boehm, using
his project experience. It is hard to regenerate these values for each new organization 
requiring effect estimation:

\bi
\item Regeneration is rarely possible due to the lack of relevant data within most organizations. 
Hence, Boehm's pragmatic advice is to reuse his effort multipliers (or the updated values
seen in~\cite{boehm00b}), then tune models using local calibration.
Note that, if this advice is followed, that the final conclusions are influenced by both 
local calibration {\em and} the values of Figure~\ref{PreciseEffortMultipliers}.
\item In the one case where we had enough domain data to attempt regeneration,
we found the same problem discussed in the introduction - dauntingly large standard deviations~\cite{me05a}. 
\ei

Happily, our ${\mu}COCOMO$ experiments show that the values of the effort estimators need 
not be too precise. Figure~\ref{ProximalEffortMultipliers} maps the precise effort multipliers 
into two approximate straight lines - one going down (for ACAP, PCAP, AEXP, MODP, TOOL, EVEP, LEXP) 
and one going up (for DATA, TURN,  VIRT, STOR, TIME, RELY, CPLX).   
{\em Proximal} value are then read from those lines and used for the effort mulitpler values.

\begin{figure}
\begin{center}
{\scriptsize
\begin{tabular}{|l|c|r@{~}|r@{~}|r@{~}|r@{~}|r@{~}|r|}
    \hline
&    &very&&&&very&extra\\
    &&low&low&nominal&high&high&high\\
    \hline
upper &ACAP           & 1.2    & 1.1& 1.00   & 0.9& 0.8     & \\
(increase &PCAP           & 1.2    & 1.1& 1.00   & 0.9& 0.8     & \\
these to&AEXP           & 1.2    & 1.1& 1.00   & 0.9& 0.8     & \\
decrease&MODP           & 1.2    & 1.1& 1.00   & 0.9& 0.8     & \\
effort)&TOOL           & 1.2    & 1.1& 1.00   & 0.9& 0.8     & \\
&VEXP           & 1.2    & 1.1& 1.00   & 0.9&          & \\
&LEXP           & 1.2    & 1.1& 1.00   & 0.9&          &\\\hline
middle&SCED     & 1.2    & 1.1& 1.00   & 1.1& 1.2     & \\\hline

lower&DATA           &        & 0.9  & 1.00   & 1.1& 1.2     & \\
(increase&TURN           &         & 0.9 & 1.00   & 1.1& 1.2     & \\
these to&VIRT           &         & 0.9 & 1.00   & 1.1& 1.2    & \\
increase&STOR           &         &     & 1.00   & 1.1& 1.2     & 1.3\\
effort)&TIME           &         &     & 1.00   & 1.1& 1.2     & 1.3\\
&RELY      & 0.8    & 0.9  & 1.00   & 1.1& 1.2     & \\
&CPLX           & 0.8    & 0.9  & 1.00   & 1.1& 1.2     & 1.3\\    \hline
\end{tabular}}
\end{center}
\caption{Proximal values for Figure~\protect\ref{EffortMultipliers}.}
\label{ProximalEffortMultipliers}
\end{figure}

Figure~\ref{NumericEstimationEffects} compares the performance of LC vs. COCOMIN+LC 
using either proximal or precise effort multipliers. As before:

\bi
\item Using just LC was an inferior policy.
\item It is hard to distinguish the performance of the more elaborate technique (precise values)
from the simpler one (proximal values).
\ei

\noindent
Therefore, we assert:

\bi
\item ${\mu}COCOMO$ can use proximal, not precise, effort multipliers.
\ei

\begin{figure*}
\begin{center}
\includegraphics{NumericEstimationEffects-SdMRE}
\includegraphics{NumericEstimationEffects-MeanMRE}
\includegraphics{NumericEstimationEffects-Pred}
\end{center}
\caption{Effect of using precise or proximal effort multipliers.}
\label{NumericEstimationEffects}
\end{figure*}

%\begin{figure}[!t]
%\begin{center}
%{
%\scriptsize
%\include{Numeric-All-SdMRE}
%\include{Numeric-All-MeanMRE}
%\include{Numeric-All-Pred}
%}
%\end{center}
%\caption{Effects (SdMRE, MeanMRE, Pred) of using precise or proximal effort multipliers.}
%\label{NumericQuartileCharts}
%\end{figure}

The adequacy of proximal effort multipliers has great operational significance.
${\mu}COCOMO$ can be deployed quickly into a new organization since there is no 
need to generate specialized effort multiplier values. That is, ${\mu}COCOMO$ 
can be used {\em sooner} than standard methods.

%------------------------------------------------------------------------- 

\subsection{Linear Regression and Model Trees}
Linear regression and model trees were included in COSEEKMO originally to 
study their effect in reducing variance. Model trees (i.e. M5P) are slower than
linear regression methods and they both require more CPU time than other methods is 
COSEEKMO. The results in Figure~\ref{LinearRegressionAndModelTrees} 
clearly show that linear regression and model trees do no better 
than the winner method (proximal effort multipliers, COCOMIN, LC) and so 
${\mu}COCOMO$ should not include them.

\begin{figure*}
\begin{center}
\includegraphics{AdditionalLearners-COCOMINWrapper-Proximal-SdMRE}
\includegraphics{AdditionalLearners-COCOMINWrapper-Proximal-MeanMRE}
\includegraphics{AdditionalLearners-COCOMINWrapper-Proximal-Pred}
\end{center}
\caption{Effect of linear regression and model trees comparing to ${\mu}COCOMO$.}
\label{LinearRegressionAndModelTrees}
\end{figure*}

%, for each data set, the new method 
%performs no worse than the method used in COSEEKMO originally.
%There are other methods used for column pruning. These more complicated 
%methods suggest using different horizons (stales) when adding or removing 
%attributes to the selected subset, ranking by sorting the attributes by  
%entropy, correlation, the difference between max and min value for each 
%attribute, and many more (LIST THEM HERE). As Figure T shows, column pruning 
%without using these suggestions performs no worse than column pruning using 
%these suggestions. Therefore, the simple column pruning method suffices for 
%this experiment. It should be noted that such suggestions increase the 
%computational running time of column pruning methods and cannot compete with 
%the linear time of this new method.
%
%FIGURE (FIGURES) T GOES HERE (FROM DAN).
%
%------------------------------------------------------------------------- 

%\subsection{Relaxing COCOMO 81 Assumptions}
%In standard practice, cost estimation is done using numeric estimations that 
%are the results of numerous studies and estimations \cite{boehm81}. These 
%{\em precise} numbers can be replaced by {\em proximal} numbers which are 
%shown to perform no worse than precise numbers (Figure \ref{NumericEstimationEffects}). 
%The proximal numbers are produced using simple lines (NEED THE DETAILS). 
%This is a relaxation of COCOMO 81's modeling assumptions of using precise 
%numbers. Figure \ref{NumericEstimationEffects} compares the precise and 
%proximal numeric estimations for column pruning followed by the learner 
%(LC), which was shown to be the best combination in reducing the deviations. 
%As always, the standard practice is shown as the baseline.
%(SHOULDN'T IT BE COCOMIN WRAPPER NOW? WE ALREADY SHOWED IT TO BEAT LOCAL WRAPPER.
%IF NOT, WE NEED TO MOVE THEM AROUND).

%------------------------------------------------------------------------- 
%
%\subsection{New COSEEKMO}
%As briefly described before, COSEEKMO was used in this experiment with 
%modifications. The new row pruning method and the new column pruning 
%method (called UNNAMED) were added to the current methods. The existing 
%proximal numeric estimation method \cite{me06d} was used more extensively 
%in the experiment. (Two new learners called UNNAMED and UNNAMED (cocomost 
%and cocomin) were also added. Although they repeatedly outperformed most 
%of the current learners, they require more studies and experimentation and 
%hence they were not reported in this paper. They are available for more 
%experimentation in future).
%
%Other changes in COSEEKMO tool are the addition of a new rejection rule 
%and reordering of the rules. Since the goal of this study was reduction 
%of deviations in the results, the new rule and the new rule ordering aimed at 
%finding lower standard deviations while reporting results that are no 
%worse than what was previously reported using COSEEKMO. Current rejection 
%rules satisfy both these conditions. The revised rejection rules are shown 
%in Figure \ref{RejectionRules}. For more details about these rules, refer 
%to \cite{me06d}.
%
%\begin{figure}{\footnotesize\begin{alltt}\sffamily
%\textbf{FUNCTION} worse(x,y)
%    \textbf{IF} statisticallyDifferent(x,y) \textbf{THEN}
%        \textbf{IF} error(x) \textless error(y)                    \textbf{THEN} \textbf{RETURN} y \textbf{FI} {\em rule1}
%        \textbf{IF} error(y) \textless error(x)                    \textbf{THEN} \textbf{RETURN} x \textbf{FI} {\em rule1}
%    \textbf{ELSE}    
%        \textbf{IF} sd(x) \textless sd(y)                            \textbf{THEN} \textbf{RETURN} y \textbf{FI} {\em rule2}
%        \textbf{IF} sd(y) \textless sd(x)                            \textbf{THEN} \textbf{RETURN} x \textbf{FI} {\em rule2}
%
%        \textbf{IF} correlation(x) \textless correlation(y)  \textbf{THEN} \textbf{RETURN} x \textbf{FI} {\em rule3}
%        \textbf{IF} correlation(y) \textless correlation(x)  \textbf{THEN} \textbf{RETURN} y \textbf{FI} {\em rule3}
%
%        \textbf{IF} pred(x) \textless pred(y)                     \textbf{THEN} \textbf{RETURN} x \textbf{FI} {\em rule4}
%        \textbf{IF} pred(y) \textless pred(x)                     \textbf{THEN} \textbf{RETURN} y \textbf{FI} {\em rule4}
%
%        \textbf{IF} \(|Subset(x)|\) \textless \(|Subset(y)|\)       \textbf{THEN} \textbf{RETURN} y \textbf{FI} {\em rule5}
%        \textbf{IF} \(|Subset(y)|\) \textless \(|Subset(x)|\)       \textbf{THEN} \textbf{RETURN} x \textbf{FI} {\em rule5}
%    \textbf{FI}
%\textbf{RETURN} 0 {\em if no reason to return true}
%\end{alltt}}
%\caption{COSEEKMO's current rejection rules. 
%{\em Error} refers to MMRE. {\em Worse}'s statistically difference test 
%compares two MMREs $x$ and $y$ using a two-tailed t-test at the 95\% 
%confidence interval.}
%\label{RejectionRules}
%\end{figure}
%
%Finally, in the evaluation scripts used in COSEEKMO, a mechanism was used 
%to remove the results that could be incomplete due to possible flaws and 
%crashes in the experiment. Unintentionally, this caused some of the results 
%to be ignored. Currently, by monitoring the experiment more carefully, such 
%mechanism was discarded. This allowed more results to be reported and hence 
%better outcomes are usually present. However, it should be noted that the 
%previous results cannot be reproduced with the exact same numeric values. 
%Nonetheless, the current results follow the same trends as they were previously 
%reported.
%
%The new COSEEKMO tool\footnote{Available at: 
%\url{http://unbox.org/wisp/tags/coseekmo}} 
%is more organized and different learners, experimental methods such as 
%different column pruning methods, and many other methods can be plugged into 
%it without requiring much modifications or knowledge about other parts.
%
%%------------------------------------------------------------------------- 
%
%\Section{Related Work}
%What goes here?
%
%%------------------------------------------------------------------------- 
%
%\Section{Conclusion}
%The new COSEEKMO tool allows for more experimentation in a short time. It 
%provides numerous methods and allows the addition of other tools. More 
%experiments can be done using the new COSEEKMO and aiming to reduce deviations 
%or to improve other performance measures.
%
%Although \cite{me05c} argue that row pruning can be useful, it does not offer 
%a full analysis of the effects of row pruning (for example row pruning without 
%column pruning). Also, as described before, since the shown results are from 
%non-identical portions of data sets, it is hard to compare column pruning and 
%row pruning combinations. \cite{me05c} offered the conclusion that ``pruning 
%always improved estimation effectiveness''. However, as Figure \ref{PruningTactics} 
%shows, row pruning does not help (or hurt) the reduction of deviations (WHAT SHOULD WE SAY ABOUT THE OTHERS), hence 
%being a neutral method in reducing deviations. \cite{me05c} also showed that 
%Pred(30) results were outstandingly better by using the combination of column 
%and row pruning. This is however shown not to be true in this study.
%
%The study by Hall and Holmes \cite{hall03} clearly showed the Wrapper as the 
%best column pruning method (except for its slow running time). Therefore, 
%Local Wrapper (a Wrapper with Local Calibration as its target learner) was 
%chosen to be the column pruning method in COSEEKMO. However, it was slow and 
%complicated. The UNNAMED Wrapper is an improvement over the Local Wrapper 
%used in the previous study \cite{me06d} both in running time and simplicity 
%of its algorithm. It performs much faster and gives results no worse than the 
%Local Wrapper. 
%
%The current experiment and COSEEKMO are designed on Boehm's COCOMO 81 
%assumptions \cite{boehm81}. The results of this experiment and the previous 
%results from COSEEKMO show this fact. However, as it is shown here, certain 
%assumptions such as the numeric estimations proposed in the current COCOMO 81 
%are not required anymore. They can be replaced with less strict and more 
%flexible ones that give no worse results. (WE ONLY SHOW THAT SD-MRE is IMPROVED. WHAT 
%ABOUT OTHER ONES LIKE CORR THAT PERFORM NOT AS GOOD. CAN WE STILL SAY THE SAME THING?)
%
%Finally, it is safe to say that the current approach to deviation reduction 
%is simpler. This is true since there are fewer modeling assumptions made 
%(the case about proximal numeric estimation) and it requires a shorter 
%running time comparing to other alternatives. Also, it uses a simpler column 
%pruning method with a linear running time in terms of the input data.
%
%%------------------------------------------------------------------------- 
%
%\Section{Future Work}
%The proximal numbers were generated manually \cite{me06d} and they require 
%more studies. The simplest way is to generate different slopes, run the 
%experiment for each set of slopes, and find the best tunings (slopes) that 
%generate the proximal numbers.
%
%The new learners, UNNAMED and UNNAMED (cocomost and cocomin), briefly 
%described before, outperformed most other learners in a shorter time. 
%However, they require more experimentation to explore different performance 
%measures in comparison with other learners.
%
%Also, if data becomes available, the same experiment can be done on COCOMO-II (IS THIS OK TO SAY OR WE ALREADY HAVE THE DATA?)
%as well as non-COCOMO data. For that purpose, simple modifications in the 
%experiment will be required.
%
%Finally, evaluation methods (such as statistics and rules) used in COSEEKMO 
%were left mostly unchallenged in this experiment. Another experiment can 
%explore different and possibly better evaluation methods. This could simplify 
%cost estimation even more.
%
%\Section{External Validity}
%
%All experiments COCOMO-style parameteric. Pre-processors to LC.
%
%Experimental results only taken from the 18 subsets within $NASA93$ and $COC81$. To the best
%of our knowledge, this represents all the publically available COCOMO-style data sets. COCOMO-II secret.
%
%------------------------------------------------------------------------- 


