\documentclass[times, twocolumn]{article} 
\usepackage{cite,paper}
\usepackage{times}
\usepackage{graphicx}
\usepackage{url}
\usepackage{alltt}
\usepackage{nopageno}
\newenvironment{smallitem}{
   \setlength{\topsep}{0pt}
   \setlength{\partopsep}{0pt}
   \setlength{\parskip}{0pt}
   \begin{itemize}
   \setlength{\leftmargin}{.2in}
   \setlength{\parsep}{0pt}
   \setlength{\parskip}{0pt}
   \setlength{\itemsep}{0pt}}{\end{itemize}}

\newenvironment{smallnums}{
   \setlength{\topsep}{0pt}
   \setlength{\partopsep}{0pt}
   \setlength{\parskip}{0pt}
   \begin{enumerate}
   \setlength{\leftmargin}{.2in}
   \setlength{\parsep}{0pt}
   \setlength{\parskip}{0pt} \setlength{\itemsep}{0pt}}{\end{enumerate}}

\newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
\newcommand{\be}{\begin{smallnums}}
\newcommand{\ee}{\end{smallnums}}
\thispagestyle{empty}

%------------------------------------------------------------------------- 
\begin{document}

\title{Column Pruning Beats Stratification in Effort Estimation}

\author{
Omid Jalali, Tim Menzies, Dan Baker\\
Lane Dept. CS \& EE \\
West Virginia University, USA\\
\small{\url{ojalali@mix.wvu.edu}, \url{tim@menzies.us}}, \small{\url{dbaker6@mix.wvu.edu}}
\and
Jairus Hihn
\thanks{
The research described in this paper was carried out at West Virginia University and
the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the
National Aeronautics and Space Administration, funded by the 
NASA Software Assurance Research Program. Reference herein to any specific 
commercial product, process, or service by trade name, trademark, manufacturer, 
or otherwise, does not constitute or imply its endorsement by the United States Government.}
\thanks{
See \protect\url{http://menzies.us/pdf/07micrococomo.pdf} for an earlier draft of this paper.}\\
Jet Propulsion Laboratory,\\
Pasadena, California, USA\\
\small{\url{jairus.m.hihn@jpl.nasa.gov}}
}

\maketitle
\thispagestyle{empty}

%------------------------------------------------------------------------- 

\begin{abstract}

Local calibration combined with stratification, also known as row pruning, is a common 
technique used by cost estimation professionals to improve model performance. The results 
presented in this paper raise several serious questions concerning the benefits of row 
pruning for improving effort estimation indicating the need to rethink standard practice. 
Firstly, the mean size of improvements from row pruning appears to be relatively small 
compared to the size of the standard deviations in effort estimation data.
Secondly, the advantages of row pruning especially for the purposes of deleting spurious 
outliers can be achieved using column pruning much more effectively. Hence, we advise 
against row pruning and advocate column pruning instead.

\end{abstract}

%------------------------------------------------------------------------- 

\Section{Introduction}
Many believe that software effort models are a waste of time.
Jorgensen reports that the majority of software effort estimates 
are expert-based and are not automatically generated from models~\cite{jorg04}. 
Effort estimation requires generalizing from a small number of old projects. 
Generalization from such limited experience is an inherently under-constrained 
problem. Hence, the learned effort models can exhibit large deviations.
For example, we reported at PROMISE 2006 that the standard deviation
on the mean {\em magnitude of the relative error}, or MRE\footnote{MRE = $abs(predicted - actual)/actual$.},
of effort estimations in 28 data sets was enormous:
\[{
\scriptsize
\{min\%,median\%,max\%\} = \{45\%, 157\%, 649\%\}
}\]
Such large deviations confuse both the generated estimates and attempts
to compare one effort estimation method with another. For example, 
Chulani \& Boehm~\cite{chulani99} and Shepperd \& Schofield~\cite{shepperd97}
report that {\em row pruning}, also known as {\em stratification}, of training data 
into related subsets improved estimation PRED\footnote{PRED(N) is the percent 
of test cases with an MRE less than N\%.} by 5\% to 12\% and 4\% to 44\% (respectively).
When compared to standard deviations in the range 45\% to 649\%, the
estimation deltas reported by Chulani \& Boehm and Shepperd \& Schofield may 
be statistically insignificant.

Regardless of their drawbacks, various institutions demand that managers produce 
model-based estimates:

\bi
\item When lobbying for resources, or arguing against funding cuts, model-based 
methods are useful tools to develop, certify, and defend an estimate. 
\item Models not only let us make decisions; they let us audit them as well.
Numerous American government bodies (DoD, NASA) require audit-able effort estimates.  
\item Models let us ``look before we leap''; e.g. effort models allow a manager to
explore and assess numerous alternate scenarios in the planning stages of a project.
\item Models can help train newcomers who can browse models to learn the processes of 
an organization.
\ei

Since model-based estimates are so useful, and since the deviation problem undermines 
the authority of those estimates, we have been exploring methods for reducing 
the deviation problem. Miller shows theoretically that decreasing the number of 
variables decreases the deviation of a linear model learned from those variables by 
minimizing least squares error~\cite{miller02}. That is, the fewer the variables, 
the more restrained are the model predictions. Inspired by Miller, and certain 
empirical results~\cite{hall03,kirsopp02}, we developed the COSEEKMO workbench~\cite{me06d}.
This COCOMO-based tool~\cite{boehm81,boehm00b} combines variable pruning with numerous alternate
model generation models (local calibration, linear regression, model trees). COSEEKMO 
assessed these alternatives via their mean performance as well as several other modeling 
criteria (reduced deviation, increased correlation, increased PRED(30)).
In the results consistent with Miller's theoretical prediction, the standard deviation 
on the MRE\% in 28 data sets fell dramatically:
\[{
\scriptsize
\{min, median, max\} : 
\overbrace{\{45, 157, 649\}}^{LC} \Longrightarrow
\overbrace{\{20, 39, 100\}}^{COSEEKMO}
}\]
(LC is the {\em local calibration} method that is standard practice in the COCOMO 
community.)

Since the deviation was reduced, it was possible to compare different methods and
select better alternatives. Hence, the mean MRE\% also decreased:
\[{
\scriptsize
\{min, median, max\} : 
\overbrace{\{43, 58, 188\}}^{LC} \Longrightarrow
\overbrace{\{22, 38, 64\}}^{COSEEKMO}
}\]
In summary, COSEEKMO implements an elaborate comparison between numerous methods. Thirty times,
the order of data in a training set is randomized, ten test cases are removed, and numerous 
techniques are applied to the remaining cases to build an effort estimator. This estimator is 
then applied to the ten test cases and comparative statistics are collected for each method.
Model selection heuristics are then applied to return the preferred model (see Figure~\ref{RejectionRules}). 
A completing justification of the design decisions made by COSEEKMO is beyond the scope 
of this paper. For full details, see~\cite{me06d}.

COSEEKMO is slow to run and after much experimentation, we have learned that most of 
that runtime is wasted. For example, COSEEKMO's model trees~\cite{quinlan92b} use 35\% of the 
CPU time yet rarely return the preferred model. Also, while certain methods are 
clearly inferior (e.g. LC), most of the remaining methods all have very similar 
performance.

Therefore, we are currently designing ${\mu}COCOMO$ (micro-COCOMO) - the subset of 
methods within full COSEEKMO that are the simplest to build, perform better than LC,
perform as well as full COSEEKMO, dramatically reduce the large deviations,
scale to larger problems, and which can be used early in the life cycle of an effort model.

An interesting result of COSEEKMO is that it strongly endorses Boehm's 1981 
research into COCOMO. Over the last three years, (starting with~\cite{me04h}), 
we have tried to improve COCOMO using modern data mining tools. To date, 
we have found no extension or variation that performs significantly better.   

There is only one area where we offer a clarification to Boehm's work. 
Prior results about the benefits of row pruning can no longer be uncritically 
accepted. For example, Chulani \& Boehm~\cite{chulani99}, report that row 
pruning (stratification) plus local calibration improved their mean PRED 
values by 5\% to 12\%. Given the large deviations seen in the top sd(MRE) plot of 
Figure~\ref{Neighborhoods}, such small improvements may be due to simple noise. 
We will show that based on the $NASA93$ data set, the advantages of row 
pruning (deleting of spurious outliers) can be achieved using column pruning. 
Further, in the case where only some of the variables are noisy, column pruning 
can delete part of a row while preserving the rest of the information in a row.

This report discusses experiments that assess the merits of adding pruning operators
into ${\mu}COCOMO$. The original version of COSEEKMO just pruned columns (variables).
An alternate strategy is to augment or replace column pruning with row pruning. 
The motivation for row pruning, also known as {\em stratification}, is very 
clear. In a COCOMO data set, each row is one project and data collected in 
one context (e.g. NASA flight software) may not be completely relevant to 
some other context. In theory, similar projects have less variation and so 
can be easier to calibrate. Nevertheless, we show here that in many 
situations, row pruning offers no advantage over other methods.
Hence, ${\mu}COCOMO$ will perform column pruning, but not row pruning.

%------------------------------------------------------------------------- 

\Section{Background}

COSEEKMO is based around COCOMO. The core intuition behind COCOMO is as a program 
grows in size, the development effort grows exponentially. More specifically:

\begin{displaymath}
{\footnotesize effort(person months)=a*\left(KLOC^b\right)*\left(\prod_jEM_j\right)}
\end{displaymath}

Here, $KLOC$ is thousands of delivered source instructions, $EM_j$ is one of 
Figure~\ref{EffortMultipliers} effort multipliers with precise values from 
Figure~\ref{PreciseEffortMultipliers}, and $a$ and $b$ are the parameters 
tuned by Boehm's local calibration (LC) procedure of Figure~\ref{LocalCalibrationProcedure}.

\begin{figure}[!t]
{\footnotesize\begin{alltt}\sffamily
\textbf{FUNCTION} worse(x,y)
    \textbf{IF} statisticallyDifferent(x,y) \textbf{THEN}
        \textbf{IF} error(x) \textless error(y)                    \textbf{THEN} \textbf{RETURN} y \textbf{FI} {\em rule1}
        \textbf{IF} error(y) \textless error(x)                    \textbf{THEN} \textbf{RETURN} x \textbf{FI} {\em rule1}
    \textbf{ELSE}    
        \textbf{IF} sd(x) \textless sd(y)                            \textbf{THEN} \textbf{RETURN} y \textbf{FI} {\em rule2}
        \textbf{IF} sd(y) \textless sd(x)                            \textbf{THEN} \textbf{RETURN} x \textbf{FI} {\em rule2}

        \textbf{IF} correlation(x) \textless correlation(y)  \textbf{THEN} \textbf{RETURN} x \textbf{FI} {\em rule3}
        \textbf{IF} correlation(y) \textless correlation(x)  \textbf{THEN} \textbf{RETURN} y \textbf{FI} {\em rule3}

        \textbf{IF} pred(x) \textless pred(y)                     \textbf{THEN} \textbf{RETURN} x \textbf{FI} {\em rule4}
        \textbf{IF} pred(y) \textless pred(x)                     \textbf{THEN} \textbf{RETURN} y \textbf{FI} {\em rule4}

        \textbf{IF} \(|Subset(x)|\) \textless \(|Subset(y)|\)       \textbf{THEN} \textbf{RETURN} y \textbf{FI} {\em rule5}
        \textbf{IF} \(|Subset(y)|\) \textless \(|Subset(x)|\)       \textbf{THEN} \textbf{RETURN} x \textbf{FI} {\em rule5}
    \textbf{FI}
\textbf{RETURN} 0 {\em if no reason to return true}
\end{alltt}}
\caption{COSEEKMO rejection rules (slightly updated from~\cite{me06d}). 
{\em Error} refers to MMRE. {\em Worse}'s statistically difference test 
compares two MMREs $x$ and $y$ using a two-tailed t-test at the 95\% 
confidence interval.
}
\label{RejectionRules}
\end{figure}

\begin{figure}[!t]
\begin{center}
{\scriptsize
\begin{tabular}{l|r@{:~}l|}\cline{2-3}
upper:   &ACAP&analysts capability\\
increase &PCAP&programmers capability\\
these to &AEXP&application experience\\
decrease &MODP&modern programming practices\\
effort   &TOOL&use of software tools\\
         &VEXP&virtual machine experience\\
         &LEXP&language experience\\\hline
middle   &SCED&schedule constraint\\\hline
lower:   &DATA&data base size\\
decrease &TURN&turnaround time\\
these to &VIRT&machine volatility\\
increase &STOR&main memory constraint\\
effort   &TIME&time constraint for cpu\\
         &RELY&required software reliability\\
         &CPLX&process complexity\\\cline{2-3}
\end{tabular}}
\end{center}
\caption{COCOMO 81 effort multipliers.}
\label{EffortMultipliers}
\end{figure}

\begin{figure}[!t]
\begin{center}
{\scriptsize
\begin{tabular}{|l|c|r@{~}|r@{~}|r@{~}|r@{~}|r@{~}|r|}
    \hline
&    &very&&&&very&extra\\
    &&low&low&nominal&high&high&high\\
    \hline
upper&ACAP   &1.46   &1.19   &1.00   &0.86   &0.71   &\\
(increase&PCAP   &1.42 &1.17   &1.00   &0.86   &0.70 &\\
these to&AEXP   &1.29 &1.13   &1.00   &0.91   &0.82   &\\
decrease&MODP   &1.2  &1.10 &1.00 &0.91 &0.82 &\\
effort)&TOOL   &1.24 &1.10 &1.00 &0.91 &0.83 &\\
&VEXP   &1.21 &1.10 &1.00 &0.90 &  &\\
&LEXP   &1.14 &1.07 &1.00 &0.95 &  &\\\hline
middle&SCED   &1.23 &1.08 &1.00 &1.04 &1.10 &  \\\hline
lower&DATA   &    & 0.94 &1.00 &1.08 &1.16&\\
(increase&TURN   &       &0.87   &1.00   &1.07   &1.15   &\\
these to&VIRT   &       &0.87   &1.00   &1.15   &1.30   &\\
increase&STOR   &       &       &1.00   &1.06   &1.21   &1.56\\
effort)&TIME   &  &    &1.00   &1.11   &1.30   &1.66\\
&RELY   &0.75& 0.88& 1.00 & 1.15 & 1.40&\\
&CPLX   &0.70 &0.85 &1.00 &1.15 &1.30 &1.65\\
    \hline
\end{tabular}}
\end{center}
\caption{Precise values for Figure~\protect\ref{EffortMultipliers}.}
\label{PreciseEffortMultipliers}
\end{figure}

\begin{figure}[!t]
\footnotesize
\begin{center}
\begin{tabular}{|p{3in}|}\hline
The matrix $D_{i,j}$ holds
\bi
\item the natural logs of the $KLOC$ estimates,
\item the natural logs of the actual efforts for projects $i \le~$j$~\le t$,
\item the natural logs of the cost drivers (the scale factors and effort 
multipliers) at locations $1 \le i \le 15$ (for COCOMO 81) or $1 \le i \le 22$ 
(for COCOMO-II).
\ei
The following calculation yields estimates for ``$a$'' and ``$b$'' that minimizes the sum of 
the squares of residual errors between predicted and actual efforts in a COCOMO model.

\begin{displaymath}
\begin{array}{rcl}
EAF_i& =& \sum_j^N D_{i,j}\\
a_0  & =& t \\
a_1  & =& \sum_i^t KLOC_i\\
a_2  & =& \sum_i^t (KLOC_i)^2\\
d_0  & =& \sum_i^t \left(actual_i - EAF_i\right)\\
d_1  & =& \sum_i^t \left((actual_i - EAF_i)*KLOC_i\right)\\
b  & =& (a_0d_1 - a_1*d_0)/(a_0a_2 - a_1^2)\\
a_3    & =& (a_2d_0 - a_1d_1)/(a_0a_2 - a_1^2)\\
a    & =& e^{a_3}
\end{array}
\end{displaymath}\\\hline
\end{tabular}
\end{center}
\caption{LC: local calibration (from ~\cite{boehm81}).}
\label{LocalCalibrationProcedure}
\end{figure}

We use COCOMO since, unlike other models such as PRICE-S~\cite{park88}, 
SLIM~\cite{putnam92}, or SEER-SEM~\cite{jensen83}, it is an open 
model with published data. All details were published in Boehm's 
text {\em Software Engineering Economics}~\cite{boehm81,boehm00b}.
There are two version of COCOMO: COCOMO-I and COCOMO-II. 
We base our studies on COCOMO-I data since we know of no large public 
domain COCOMO-II data sets. In contrast, several COCOMO-I data sets 
are available in the PROMISE repository~\cite{me07c}:

\bi
\item $COC81$: The 63 projects used to define COCOMO-I. This data comes 
from a variety of domains including engineering, science, financial, etc. 
\item $NASA93$: The 93 NASA projects from the 1980s and 1990s. Since this 
data comes from NASA, it is stratified to just aerospace applications.
\ei

%------------------------------------------------------------------------- 

\Section{External Validity}

The external validity of our study must be discussed. These results are only
relevant to COCOMO-style parametric effort estimation. They are based only 
on the projects described in the $COC81$ and $NASA93$ data sets. Nevertheless, 
to the best of our knowledge, $COC81$ and $NASA93$ represent the universe of 
publicly available COCOMO data sets.

Also, our results are limited to the space of learners, WRAPPERS, and other 
methods explored by COSEEKMO and ${\mu}COCOMO$, which are:

\bi
\item local calibration,
\item model trees,
\item linear regression,
\item LocalWRAPPER, and
\item Euclidean nearest neighbor.
\ei

It is possible that other methods would outperform and overturn our results. 
In the true spirit of PROMISE, all our data and tools are on-line. We invite 
other researchers to download them, extend them, and to supersede our results.

%------------------------------------------------------------------------- 

\Section{Experiments}

%------------------------------------------------------------------------- 

\subsection{Prior Row Pruning Experiments}

Our previous experiments were quite negative on the merits of row pruning.
As reported in \cite{me06d}, the rows of $NASA93$ divide into various subsets:

\bi
\item All the records, 
\item Whether or not it is flight or ground system, 
\item The parent project that includes the project expressed in this record, 
\item The NASA center where the work was conducted, 
\item etc. 
\ei

Some of these subsets are bigger than others. Given a record labeled with subsets 
$\{S_1, S_2, S_3 , . . .\}$ then we say a {\em candidate stratification} has 
several properties: 
\bi
\item It contains records labeled $\{S_i, S_j\}$. 
\item The number of records in $S_j$ is greater than $S_i$; i.e. $S_j$ 
is the superset and $S_i$ is the stratification. 
\item $S_j$ contains 150\% (or more) of records in $S_i$. 
\item $S_i$ contains at least 20 records. 
\ei

COSEEKMO searched for effort models that were better in the subset than the superset.
$NASA93$ contains 207 candidate stratifications (including one stratification 
containing all the records), and better effort models were found in only four cases.
That is, in the overwhelming majority of cases, row pruning (stratification) offered
no advantage.

%------------------------------------------------------------------------- 

\subsection{New Row Pruning Experiments}

Another possibility, not previously explored, is that the stratifications are mis-labeled.
The stratifications seen in $COC81$ and $NASA93$ come from row labels. If domain experts 
do not fully understand the factors that affect software costs, they might inadvertently group 
together the wrong rows.

To test that possibility, we performed row pruning experiments where stratifications 
were selected from the training set via their Euclidean distance to the test case. 
That is, for each member of the test set, an effort model is generated (using LC) 
from the closest $N$ members of the training set. For this experiment, $distance$ was 
defined using:
\[{
\scriptsize
\mbox{distance case1 to case2} = \sqrt{\sum_{X \in EM} \left(\frac{X_1 - X_2}{max(X) - min(X)}\right)^2}
}\]
(where the X values are the numeric values of Figure~\ref{PreciseEffortMultipliers}).

Figure \ref{PruningTactics} shows the effects of different combinations of row and column
pruning on the standard deviation of the MRE, the mean MRE, and PRED(30).
This is our standard report format so, before reviewing those findings, we review
the details of those reports:

\bi
\item The x-axis of Figure~\ref{PruningTactics} denotes which subset of $COC81$ or $NASA93$ 
was used in this experiment.  
\item Plots are always shown in sets of three. A vertical line through the three plots would
show results from the same subset.
\item The plots were generated using the COSEEKMO method described in the introduction. 
\item For $sd(MRE)$ and $mean(MRE)$, the goal is to get $lower$ lines than the other methods.
\item For $\%pred(30)$, the goal is to  get $higher$ lines than the other methods.
\ei

\begin{figure*}[!t]
\begin{center}
\includegraphics{PruningTactics-SdMRE}
\includegraphics{PruningTactics-MeanMRE}
\includegraphics{PruningTactics-Pred}
\end{center}
\caption{Effects of different pruning tactics.}
\label{PruningTactics}
\end{figure*}

\begin{figure*}[!t]
\begin{center}
\includegraphics{LocalWrapper-Precise-LOCOMO-SdMRE}
\includegraphics{LocalWrapper-Precise-LOCOMO-MeanMRE}
\includegraphics{LocalWrapper-Precise-LOCOMO-Pred}
\end{center}
\caption{Effects of different neighborhoods.}
\label{Neighborhoods}
\end{figure*}
 
\noindent
As to the particulars of Figure~\ref{PruningTactics}: 

\bi
\item These plots show results from some preprocessing of the data followed 
by local calibration (LC). In the case of Figure~\ref{PruningTactics}, those 
preprocessors were some combination of row and column pruning. 
\item In this plot, LC used the precise effort multiplier values of Figure~\ref{PreciseEffortMultipliers}.
\item The lines labeled ``Precise-None-None-LC'' show the standard effort 
estimation method as recommended by Boehm; i.e. just perform local calibration.
\item The top two lines of each plot show a comparison between row pruning 
and no row pruning when there is no column pruning present.
\item The last two lines add in column pruning. For this figure, the column pruning method
was based on Kohavi's WRAPPER~\cite{kohavi97}. The WRAPPER passes different subsets of
variables to some oracle (in our case, LC) and returns the subset which yields the best 
performance. WRAPPER is thorough but can be slow since (in the worst case), it has to 
explore all subsets of the available variables.
\item The row pruning in these results used the $N=20$ closest cases to build an effort model 
for the test cases (and the impact of changing $N$ is discussed below).
\ei

In Figure~\ref{PruningTactics}, the following effect is a strong argument that local 
calibration, by itself, needs to be augmented or replaced: 

\bi
\item The ``Precise-None-None-LC'' results in the top $sd(MRE)$ plot show devastating 
large deviations resulting from standard methods (LC) - often over 100\% and sometimes
much larger.
\ei

The following effects are strong arguments against row pruning via Euclidean distance measures:

\bi
\item The top two plots for $sd(MRE)$ and $mean(MRE)$ show that without other 
pruning methods, row pruning does no better than LC.
\item The top two plots also show that when column pruning is active, row pruning 
adds little or no advantage.
\ei

The following effects are strong arguments for column pruning:

\bi
\item The top two plots show that the column pruning always resulted in much less error and deviation.
\item In the middle plot, column pruning without row pruning generates the smallest errors.
\ei

The only argument that can be mounted from these results against column pruning is:

\bi
\item In the bottom plot, column pruning, by itself, sometimes yields the lowest PRED.
This effect is much less marked than those discussed above. So, it does not dissuade
against our thesis that there is much merit in column pruning and little merit in row pruning.
\ei

These results run counter to several other publications 
(e.g.~\cite{chulani99,shepperd97}) including our research~\cite{me04h,me05c}
that report an improvement in PRED following stratification. One possibility 
is that the above results did not select the right number of neighbors to build an 
effort estimation model.

Figure~\ref{Neighborhoods} tests that speculation. The proceeding discussion endorsed 
column pruning with the WRAPPER followed by LC. Figure~\ref{Neighborhoods} augments 
such a rig with a nearest-neighbor section of \mbox{$N \in\{5,10,20,40,80\}$}. 
No clear ``best'' nearest-neighbor is apparent. Indeed, looking at the mean(MRE) 
results, changing the size of neighborhood had very little effect on the mean error.
All in all, there is no evidence in Figure~\ref{Neighborhoods} that the above results 
are conflated by a wrong selection of nearest neighbors. 

%------------------------------------------------------------------------- 

\Section{Discussion}

\noindent
Based on the above, we assert that:

\be
\item Column pruning is particularly useful at reducing the mean and standard deviation
of the MRE seen with COCOMO-style local calibration. This effect has been reported previously~\cite{me06d}.
\item Row pruning offers no advantage for the generation of effort models using a COCOMO-style 
LC procedure. This is a new result.
\ee

Conclusion \#2 is counter-intuitive. Many researchers (e.g.~\cite{chulani99,shepperd97,me04h,me05c})
advise that a standard practice for modeling is to ``remove outliers''; i.e. cull rows of 
data that contain spurious signals. We speculate that prior results regarding the merits of 
stratification may have been conflated by the large deviation effect; i.e. their reported improvements 
were merely noise within a widely variant space of solutions.

Another possibility is that the ``remove outliers'' advice is subtly misguided. In the 
case where the outliers are caused by a small number of deviant variables, then if those 
variables are removed (by column pruning), the information in the rest of the row remains 
and can assist in the modeling task. In that case, the ``remove outlier'' advantage seen 
in row pruning would also be achieved with column pruning.

Our own prior results that endorsed row pruning need further discussion. 
Chen, Menzies, Port, and Boehm (hereafter, CMPB)~\cite{me05c} concluded that ``pruning always 
improved estimation effectiveness''. That conclusion was made based on the results 
in Figure~\ref{OtherPapersResults}. The y-axis of that figure shows the PRED(30) results seen
after performing 30 linear regressions on $\frac{2}{3}$-rds of the data (random selected),
then applying the learned model to the remaining $\frac{1}{3}$-rd of the data.
The top $before$ plot shows the linear regression results while the $after$ plot shows 
the effect of linear regression plus column pruning via the WRAPPER. The data sets on the 
x-axis are sorted left-to-right according to how many records they contain (see the blue plot). 
To produce those data sets, the four data sets shown on the left hand side 
($cii0,cii4,coci,na60$) were divided two ways:

\be
\item The data was divided into the project, geographical center, and task performed
by the software. This generated the three middle data sets ($pall,call,tall$).
\item The data was divided according to their most specific stratification; e.g.
a project from a particular project at a particular center working on a particular
task. This generated the eight smallest data sets shown on the right 
($c03,c01,t03,p04,t02,c02,p02,p03)$.
\ee

That is, the data sets in the middle and on the right of Figure~\ref{OtherPapersResults} 
are stratifications (row prunings) of data shown on the left. 

From those results, CMPB concluded a column pruning, plus extensive row pruning, was the best 
effort estimation policy. However, those results applied row and column pruning to non-identical 
portions of data sets when comparing different combinations of column and row pruning.

In contrast, the analysis discussed here compares row pruning separately and in combination with column 
pruning. It also uses identical data sets in all these comparisons. Hence, this paper is a more 
complete analysis of row pruning and column pruning effects.

\begin{figure}[!t]
\begin{center}
\includegraphics[width=3.2in,angle=270]{colp}\includegraphics[width=3.2in,angle=270]{lightRowColp}\includegraphics[width=3.2in,angle=270]{heavyRowColp}
\end{center}
\caption{Results from~\cite{me05c}.}
\label{OtherPapersResults}
\end{figure}

%------------------------------------------------------------------------- 

\Section{Conclusion}

From our results, we can make several comments about certain prior results~\cite{me05c,me06d}.

%------------------------------------------------------------------------- 

\subsection{Chen, Menzies, Port, \& Boehm '05~\cite{me05c}} 

The estimation improvements attributed to row pruning by Chen et.al.~\cite{me05c} were not found here.
Those results were due to a methodological error: i.e. row and column pruning was not applied 
consistently to that data. Here, after applying column and row pruning to the same data,
we found that row pruning gave no improvement over column pruning.

%------------------------------------------------------------------------- 

\subsection{Menzies, Chen, Hihn, \& Lum '06~\cite{me06d}}

What do these results say about the COSEEKMO study~\cite{me06d}? Is it a needlessly elaborate system?

We would argue for the utility of COSEEKMO. COSEEKMO is a {\em certification} environment and ${\mu}COCOMO$ 
is a {\em production} environment. COSEEKMO is a general rig where new ideas and tools can be 
rigorously and exhaustively tested. However, it is also the flame that burns away the fat. 
Using COSEEKMO, we can find lean and fast methods for effort estimation, which can simplify the 
work of analysts performing model-based estimation. Our current plan is to code all those lean and fast 
methods into ${\mu}COCOMO$.

The design process of ${\mu}COCOMO$ imposed an audit of effort modeling methods. Such an audit 
is long overdue. Many effort estimation methods such as clustering~\cite{garre05}, 
neural networks~\cite{mair00investigation}, and case-based reasoning~\cite{shepperd97} 
exist in the literature. In industry, the most commonly-used formal techniques are 
parametric or regression-based techniques, as used in COCOMO-I~\cite{boehm81}, 
COCOMO-II~\cite{boehm00b}, SLIM~\cite{putnam92}, SEER-SEM~\cite{jensen83}, and PRICE-S~\cite{park88}.
There are very few published studies that empirically compare this diverse set. 
What is more usual are narrowly focused studies, such as those conducted by 
Kemerer~\cite{kemerer87}, Briand~\cite{briand00}, Lum~\cite{lum02}, or 
Ferens~\cite{ferens98} that test linear regression models in different environments.

The reason for this lack of empirical studies is two-fold. As mentioned in the 
introduction, most effort estimation is expert-based, not model-based, and 
expert-based evaluations are difficult to repeat. Also, the large deviation problem 
makes it very difficult to come to definitive conclusions about the relative
merits of different model-based methods. 

COSEEKMO's ability to reduce that variance should change the nature of empirical 
effort estimation research. It is now possible to compare, and reject, certain 
model-based methods as (i)~inferior or (ii)~competitive in terms of MRE or PRED, 
but inferior in terms of implementation complexity or runtime.

We therefore anticipate a few years of COSEEKMO/${\mu}COCOMO$-style experimentation 
where model-based methods are debated before some consensus emerges on best-practices 
for model-based effort estimation. Once that consensus is available, the field 
can move on to comparing our best model-based methods with expert-based estimation.

%------------------------------------------------------------------------- 

\Section{Future Work}

It is possible that stratification to particular project types would improve 
effort estimation. What we have shown in our work so far is that, to date, 
we have not found that the current COCOMO attributes can characterize those
``particular project types''. It is possible that further details from the 
target domain could be used to improve effort estimation; e.g. when
Boetticher augmented the standard effort estimation attributes with knowledge 
of user-interface complexity, the resulting models had impressively large 
PRED values~\cite{boetticher03}. 

Therefore, over summer 2007, we plan to experiment with augmented COCOMO 
data sets that include more domain-specific descriptors of projects. 
The hypothesis is that the information in these additional columns will 
improve effort estimation; i.e. the row pruning based on these extra 
columns will decrease MRE and increase PRED.

In addition to the above, the authors of this paper are exploring other
measures, such as non-parametric methods to evaluate the experiment results,
to check and verify the stability of these conclusions.

\bibliographystyle{paper}
\bibliography{refs}
\end{document}

%------------------------------------------------------------------------- 
