
\section{Change}

For these reasons, we study effort estimation models and how to improve them.
\begin{figure}[!t]
\begin{center}
\includegraphics[width=3.2in]{means.pdf}
\end{center}
\caption{The effect of removing attributes selected by FSS.}\label{fig:mean}
\end{figure}
To the best of our knowledge, this is the first report of applying {\em feature subset selection} (FSS)
to software effort  data. Our  results,
shown in \fig{mean}, show that FSS can dramatically improve cost estimation.  The x-axis of that figure
shows five ``cut sets'', each containing several attributes (this notion of a ``cut set''
is explained in \S4.2). These cut sets were discovered and ordered using FSS. Each  x-axis
point shows the effects of removing all the attributes in all the cut sets up to that point.
``Cut0'' shows the baseline results using all available  attributes (i.e. no cuts).   
In all the five data sets studied in \fig{mean}, discarding at least the first three cut
sets {\em always} lead to  a statistically significant improvement.
Sometimes, the improvement was quite striking;  e.g. ``p02'''s performance
rises from just over 20\% to nearly 100\%. 

The core technology used in this study is FSS (feature subset selection) and
FSS  is an efficient heuristic search through
subsets of the available attributes. The goal of this search is to
find a subset that gives similar, if not superior, performance than
using all the attributes.  \eq{one01} demonstrates how large that
space can be.  There are 15 parameters except for SIZE (total 16
parameters in COCOMO 81). 

The rest of this paper explains and expands the \fig{mean} result.
We begin with some introductory notes on feature subset selection and the COCOMO
effort estimation model  used in this study.
This is followed by some notes on our experimental design and results.
T-tests are applied to the results to demonstrate that always in our data sets,
removing attributes improves performance {\em without} increasing the variance
in model behavior.


\begin{figure*}
\begin{center}
{\scriptsize
\begin{tabular}{|r|c|r||r|c|r||r|c|r||r|c|r||r|c|r|}
\hline \multicolumn{3}{|c||}{COCOMO81} &\multicolumn{3}{c||}{NASA}
&\multicolumn{3}{c||}{Project02} &\multicolumn{3}{c||}{Project03}
 &\multicolumn{3}{c|}{Project04} \\
\hline \multicolumn{3}{|c||}{instances: 63} &\multicolumn{3}{c||}{instances: 60}
&\multicolumn{3}{c||}{instances: 22} &\multicolumn{3}{c||}{instances: 12}
 &\multicolumn{3}{c|}{instances: 14} \\
 \hline
 cut &attr. &n &cut &attr. &n &cut &attr. &n &cut &attr. &n &cut &attr.
 &n\\
 \hline\hline
 6 &loc&10 &6 &loc &10 &6 &loc &10 &6 &loc &10 &6 &loc &10\\
 \hline
 5 &sced &10 &5 &acap &10 &5 &turn &7 &5 &modp &9 &5 &modp &9\\
 \hline
 5 &pcap &10 &5 &time &10 &5 &lexp &7 &4 &pcap &1 &5 &virt &5\\
 \hline
 5 &time &10 &5 &turn &10 &4 &time &4 &4 &rely &1 &4 &stor &4\\
 \hline
 5 &virt &9 &4 &stor &8 &4 &modp &3 &4 &turn &1 &4 &turn &4\\
 \hline
 4 &cplx &6 &4 &vexp &7 &3 &data &2 &3 &stor &0 &4 &cplx &3\\
 \hline
 4 &modp &5 &3 &data &4 &3 &tool &2 &2 &acap &0 &3 &time &2\\
 \hline
 4 &acap &5 &3 &aexp &2 &3 &sced &2 &2 &vexp &0 &3 &pcap &2\\
 \hline
 4 &rely &5 &2 &virt &1 &2 &rely &1 &1 &aexp &0 &3 &rely &2\\
 \hline
 3 &vexp &4 &2 &pcap &1 &2 &vexp &1 &1 &data &0 &3 &lexp &2\\
 \hline
 3 &tool &3 &2 &modp &1 &2 &cplx &1 &1 &tool &0 &2 &acap &1\\
 \hline
 3 &data &3 &1 &cplx &0 &1 &aexp &0 &1 &cplx &0 &2 &data &1\\
 \hline
 2 &aexp &2 &1 &tool &0 &1 &pcap &0 &1 &lexp &0 &2 &vexp &1\\
 \hline
 2 &stor &2 &1 &rely &0 &1 &virt &0 &1 &sced &0 &2 &tool &1\\
 \hline
 1 &lexp &1 &1 &lexp &0 &1 &acap &0 &1 &virt &0 &2 &sced &1\\
 \hline
 1 &turn &0 &1 &sced &0 &1 &stor &0 &1 &time &0 &1 &aexp &0\\
\hline
\end{tabular}
}
\end{center}
  \caption{Features selected in each data set.}
   \label{fig:Grouping}
\end{figure*}


%\vspace{6 pt}

\section{Feature Subset Selection}


%\vspace{6 pt}

Feature subset selection is the process of identifying the most
promising features in a given dataset. Datasets used in practical data
mining applications have a large number of features. These data sets
often contain several extraneous features which can reduce the
efficiency of the learning algorithm. Feature subset selection helps
us identify the important attributes and remove redundant ones. If
only the most relevant features were to be selected and given to the
learning algorithm they can produce smaller theories. This enhances
the understanding of the dataset or domain under consideration.
Dimensionality reduction also speeds up the learning process. Also, a
repeated result in the FSS field (e.g.~\cite{hall03}), is that
ignoring features need not degrade the performance of the learned
theory.

%\vspace{6 pt}


In this study, we applied the WRAPPER FSS method implemented in WEKA\footnote{
The WEKA data mining toolkit is free, open source, well documented,compatible on many platforms, and easy to install.} data mining toolkit ~\cite{witten99}.  When using WRAPPER,
a target learner is augmented with a pre-processor that used a
heuristic {\em forward select} search to grow subsets of the available
features. At each step in the growth, the target learner is called to
determine the performance of the model learned from the current
subset. Subset growth is stopped when the growth is {\em stale};
i.e. after a {\em MAX\_STALE} number of times, adding attributes has not improved
the performance.

For example, suppose the set of attributes were \{A,B,C, D,E,F,...,Z\} and MAX\_STALE was 2. 
WRAPPER
starts by selecting one attribute at random (e.g. C) and score its performance.
\[<Selected=\{C\},  Score=30, Stale=0>\]
Next, another  randomly selected attribute (e.g. B) is added and scored:
\[<Selected=\{C,B\},Score=50, Stale=0>\]
Note how the addition of B was not a {\em stale} addition since it improved the score. However,
the addition of the next randomly selected attribute (e.g. E) does {\em not} improve the score,
so {\em stale} increments:
\[<Selected=\{C,B,E\},  Score=40, Stale=1>\]
Similarly, adding D also fails to improve the score beyond just using \{C,B\} so scale
increments again.
\[<Selected=\{C,B,E,D\},  Score=42, Stale=2>\]
Since MAX\_STALE has been reached, WRAPPER would remove  from the {\em selected}
set all the attributes implicated
in the stale growth (\{E,D\}). The search would then continue, using other attributes.

One of the major advantages of the WRAPPER approach is that, if some
target learner is already implemented, then the WRAPPER is simple to
implement.  Also, in their comparative evaluation of feature subset
selection techniques~\cite{hall03}, Hall and Holmes conclude that
WRAPPER is the best FSS mechanism, if the data set is not too large.
At each step in the heuristic search, WRAPPER makes another call to
the target learner. Hence, it many be too slow for large data sets.
%\vspace{6 pt}
The
data sets used in this study, are small (maximum size: 
 63 instances) and hence are amenable for WRAPPER.

The Hall and Holmes results were also negative about a widely used
technique: principle component analysis (PCA). FSS methods can be
grouped according to:
\bi
\item
Whether or not they make special use of the
target attribute in the data set such as ``development cost'';
\item
Whether
or not they use the target learner as part of their FSS analysis.
\ei
PCA
is unique since, unlike other FSS methods, it {\em does not} make
special use of the target attribute. WRAPPER is also unique, but for
different reasons: unlike other FSS methods, it {\em does} use the
target learner as part of the FSS analysis. Hall and Holmes found that
PCA was one of the worst performing FSS methods (perhaps because it
ignored the target attribute) while WRAPPER was the best (since it can
exploit its special knowledge of the target learner).



\section{Experimental Design}


%\vspace{6 pt}

Our experiments were in three phases: {\em linearization},
followed by {\em selection} followed by 
{\em application}. Each phase was applied to our five data sets.

\subsection{Linearization}

%\vspace{6 pt}


{\em Linearization} was described above: all symbols are replaced with the logarithm
of their effort multiplier value.

\subsection{Selection}

%\vspace{6 pt}

In {\em selection}, a data set is randomly sub-sampled ten times to
generated ten random samples, each containing
90\% of the data. WRAPPER is then applied to each of the ten sub-samples
and the selected attributes were recorded.  ``Cut
sets'' are then formed by clustering the attributes
according to how often they were selected.

%\vspace{6 pt}


The {\em selection} results are shown in \fig{Grouping}.  In that
figure, attributes are sorted but how often they appear (see the
results in the $n$ column).  For example, in COCOMO-81, {\em loc}
(lines of code) was selected in all 10 subsets while {\em turn}
(turnaround time) was never selected. 

%\vspace{6 pt}


COCOMO-81's cut sets are shown in the far left-hand-side column
(labeled $cut$).  $Loc$ has a special place in the COCOMO model
(recall~\eq{two}) so it is allocated a cut set all to itself. The
other attributes are grouped into five cut sets with the least/most
frequently occurring attributes being assigned to cut sets 1/5
(respectively).   

\subsection{Application} 

%\vspace{6 pt}


Each selection phase for each data sets resulted in cut sets
specialized to that data set. In the {\em application} phase, the cut
sets found for each data set were explored as follows.  The attributes
in each cut set was removed in the order 1,2,3,4,5 (as defined in
\fig{Grouping}). After the removal of each attribute set, the
remaining data was randomly sampled thirty times to generate a
$\frac{2}{3}$rds training set and a $\frac{1}{3}$rds test set.
Least squares regression was then applied to the training set
to learn a  linear model of the form of \eq{two}. This model was then
applied to the test set. The mean and standard deviation 
of the model's performance over the 30 subsets was then computed. 

\vspace{10 pt}

\section{Results}\label{sec:perform}

%XXX table numbers shown graphically in  figure X
%XXX  sd numbers shown graohcailly in figure Y


\newcommand{\good}{}


\fig{meanpred30} shows the raw data used to plot the mean PRED(30)
results shown in \fig{mean}.  Each column shows the results after
removing some combination of the cut sets. The final column in that
table shows the effects of removing all attributes except lines of
code.  In that figure, ``*'' denotes mean values that are significantly
different (at the 99\% level) to the $cut0$'s mean. 
In \fig{meanpred30}, the {\em maximum mean PRED(30)} values
found for any data set are shown underlined and in bold.  Note that:
\begin{figure}[!t]
{\scriptsize
\begin{center}
\begin{tabular}{|r@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c|}\hline
Cut What?     &     cut0&     cut1 &     cut1,2 &     cut1,2,3 &  cut1,2,3,4    &       justLOC\\\hline
 dataset       &  keep1-6   &   keep2-6 &   keep3-6  &   keep4-6 & keep5-6  &   justLOC\\\hline
 cocomo81    &  42.4  & 46.5 {\good}* & {\bf  \underline{51.3}} {\good}*  & 41.6 ~~    & 47.3 {\good}* & 22.1 {\good}* \\
      nasa60        &  69.5  & 74.2 {\good}* & 78.0 {\good}*  & {\bf \underline{81.3}} {\good}*  & 80.5 {\good}*  & 57.3 {\good}* \\
project04    &   27.5 & 20.8 {\good}*  & 16.7 {\good}*  & {\bf \underline{66.7}} {\good}*  & 57.5 {\good}*  & 62.5 {\good}* \\
     project03   &  20.0    & 19.2 ~~    & 54.2 {\good}*  & {\bf \underline{55.8}} {\good}*  & 54.2 {\good}*  & 36.7 {\good}* \\
     project02   &  15.7  & 84.8 {\good}*  & 94.8 {\good}*  & {\bf \underline{97.1}} {\good}*  & 92.9 {\good}*  & 85.2 {\good}* \\\hline
    MEANs  & 35 ~~   & 49.1 ~~    & 59.0  ~~     & {\bf \underline{68.5}} ~~    & 66.5 ~~    & 52.8 ~~ \\\hline
\end{tabular}
\end{center}}
\caption{Mean PRED(30) results}\label{fig:meanpred30}
\end{figure}                     



\begin{figure}[!t]
\begin{center}
{\scriptsize
\begin{tabular}{|r@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c|}\hline
Cut What?       &  cut0   &   cut1 &   cut1,2  &   cut1,2,3 & cut1,2,3,4  &    justLOC\\\hline
 dataset       &  keep1-6   &   keep2-6 &   keep3-6  &   keep4-6 & keep5-6  &   justLOC\\\hline
 cocomo81      & 11.75   &  9.66   &  8.17  & 8.10 & 8.92 & {\bf \underline{7.66}} \\
   nasa60      &  8.55   &  8.21   &  8.47  & 7.54 & {\bf \underline{7.32}} & 8.58 \\
  project04    & 12.07   &12.40    &  8.78  & {\bf \underline{6.92}} &11.10 &10.93 \\
   project03   & 20.13   &20.43     & 18.67 &{\bf \underline{15.65}} &19.79 &22.49 \\
   project02   & 23.07   &17.47     & {\bf \underline{15.16}} &23.06 &23.81 &26.06 \\\hline
\end{tabular}}
\end{center}
\caption{Pred(30)s standard deviations}\label{fig:variancetable}
\end{figure}
\bi
\item
The $cut0$ means were always  lower than means seen in the other cut sets. 
\item
The best (i.e.) largest  mean values occurred after removing up to and including
cut set 3.
\ei

\fig{variancetable} shows the raw data used to plot the standard
deviation results shown in \fig{variance}.  
As before, each column shows the results after
removing some combination of the cut sets and the final column 
shows the effects of removing all attributes except lines of
code. 
In \fig{variancetable},
the {\em minimum PRED(30) standard deviation} values are shown 
underlined and in bold. Note that:
\bi
\item As above, the most best (i.e. smallest)
standard deviations occurred after removing up to and including cuts set 3.
\ei

%\newcommand{\great}{\textbullet}

%\newcommand{\great}{\textbullet}


%\begin{figure}[!t]
%{\scriptsize
%\begin{center}
%\begin{tabular}{|r@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c|}\hline
% dataset     &     cut0&     cut1 &     cut1,2 &     cut1,2,3 &  cut1,2,3,4    &       justLOC\\\hline
% cocomo81    &  42.4  & 46.5 {\good}{\great} & {\bf  \underline{51.3}} {\good}{\great}  & 41.6 ~~    & 47.3 {\good}{\great} & 22.1 {\good}{\great} \\
%      nasa60        &  69.5  & 74.2 {\good}{\great} & 78.0 {\good}{\great}  & {\bf \underline{81.3}} {\good}{\great}  & 80.5 {\good}{\great}  & 57.3 {\good}{\great} \\
%project04    &   27.5 & 20.8 {\good}{\great}  & 16.7 {\good}{\great}  & {\bf \underline{66.7}} {\good}{\great}  & 57.5 {\good}{\great}  & 62.5 {\good}{\great} \\
%     project03   &  20.0    & 19.2 ~~    & 54.2 {\good}{\great}  & {\bf \underline{55.8}} {\good}{\great}  & 54.2 {\good}{\great}  & 36.7 {\good}{\great} \\
%     project02   &  15.7  & 84.8 {\good}{\great}  & 94.8 {\good}{\great}  & {\bf \underline{97.1}} {\good}{\great}  & 92.9 {\good}{\great}  & 85.2 {\good}{\great} \\\hline
%    MEANs  & 35 ~~   & 49.1 ~~    & 59.0  ~~     & {\bf \underline{68.5}} ~~    & 66.5 ~~    & 52.8 ~~ \\\hline
%\end{tabular}
%\end{center}}
%\caption{Mean PRED(30) results}\label{fig:meanpred30}
%\end{figure}                     



\begin{figure}[!t]
\begin{center}
\includegraphics[width=2.5in]{sds.pdf}
\end{center}
\caption{The changes in variance with feature selection.}\label{fig:variance}
\end{figure}


%\vspace{6 pt}

The data was studied using t-test comparing the mean PRED(30) results
for each data set after removing sets $X_0$=cut0; $X_1$=cut1;
$X_2$=cut2; $X_3$=cut3; $X_4$=cut4; $X_5$=cut everything except lines
of code.  Our experimental rig let us compare results 30 train/test
results seen between sets $X_i$ and $X_j$ where $\{i,j\}\in \{0...5\}$ and $j<i$. If the
T-tests reject the hypotheses that there is no difference in the
difference between the $<X_i,X_j>$ comparison, then that is scored as one
``tie''.  If the comparison does not ``tie'', then if the means are
numerically compared to compute ``win''s and ``loss''es. 


%\vspace{6 pt}

The results,
sorted on {\em win-loss} are shown in \fig{wins_losses}\footnote{There are 6 feature sets, 5 data sets. Each feature set needs to be compared with other 5 feature sets for each data set, so the total number of comparisons for each feature set is 5x5=25.}.  Note that
culling up to cut set 3 and 4 always generated data sets which, when
measured in terms of {\em win-loss}, score better than other
combinations of attributes.


%\vspace{6 pt}

Recall from \fig{meanpred30} and \fig{variancetable} that the maximum
mean and minimum standard deviations were seen in {\em cut3}. Hence,
even though {\em cut4} sets scores as well
as {\em cut3} in \fig{wins_losses}, we recommend cutting up to $cut3$,
but not $cut4$. 

%\vspace{6 pt}


At $cut3$, \fig{variance} shows that the variances were never decreased
by removing attributes and sometimes it even decreased.
Hence, in summary, the above results are quite positive about feature
subset selection for software cost estimation.  Not only do the PRED(30) means
significantly increase {\em without} also increasing the standard deviations.

%% \section{External Validity}

%% In order to use the linear least squares regression, we transform
%% COCOMO model -- exponent model into a linear model. There will be
%% two threats to our study: one is linearity and the other is
%% granularity. The linearity assumption - the region between two
%% samples changes smoothly between the two samples, will not be hold
%% in some regions where the performance of the exponent model changes
%% radically in some space of the calibration exponent parameter b.
%% Granularity is a related concern: if a model is searched in steps of
%% size M, and the model's behavior can change dramatically is some
%% space N $\le $ M, then the explorer can miss important features of
%% the model.

%% Even there exists some space where the performance of the model
%% changes dramatically, after the transformation from the exponent
%% model into the linear model, that changes have already been reduced
%% a lot. Furthermore, our experiments complete search through the
%% range of exponent parameter b values makes no extrapolation
%% assumptions. Hence, our results are immune to linearity problems.

%% We explore different data sets -- COCOMO81 and NASA, and different
%% data sub-sets in NASA. All results from the different data sets
%% clearly support our conclusions. Hence, we are somewhat confident
%% that our experiments do not have a granularity problem.

%% We randomly select the training set and test set, repeat the
%% experiment 30 times, and avoid the threat of order effects. To avoid
%% the chance of over-fitting, we conduct holdout studies. With the
%% experiments on different data sets, we are reducing the odds that
%% our results arise from some statistical quirk.


\begin{figure}[!t]
{\scriptsize
\begin{center}
\begin{tabular}{|r|cccc|}\hline
treatment           & Win-Loss   &  Win   &  Loss  &  Tie\\\hline
cut4      &  13 & 16 &  3 & 6 \\
cut3      &  13 & 16 &  3 & 6\\
cut2      &   8 & 14 &  6 & 5 \\
cut1      &  -9 &  6 & 15 & 4 \\
justLOC    & -10 &  6 & 16 & 3 \\
cut0      & -15 &  4 & 19 & 2\\\hline
\end{tabular}
\end{center}
}
\caption{Wins vs Losses}\label{fig:wins_losses}
\end{figure}

\section{Discussion}


%\vspace{6 pt}


A curious feature of 
\fig{Grouping} is that, with the exception of lines of code,
different data sets yield different ``most important'' attributes. 
Hence, it would be inappropriate to use these results of argue that we
should always ignore certain COCOMO attributes. Indeed, our results
seem to endorse the opposite view: analysts should collect as many different
attributes as possible {\em before} applying FSS to select the attributes
that are most important in a particular domains.


%\vspace{6 pt}


From a practical standpoint, removing cost-drivers variables produces a more efficient cost model turned to an organization��s existing project practices. However, if the organization makes a significant change in practices with respect to a removed cost driver variable (for example, going from developing routine business applications to developing real-time, safety-critical systems), there will be a significant risk of estimation inaccuracy. We propose an idea for the extended reduced-parameter model: Extended Effort = Reduced-Parameter Effort * Reduced-But-Current-Significant Parameter. 
We use the extended reduced-parameter models to answer such a question: what happen if the organization uses the reduced-parameter model without RELY to bid a very-high-Reliability project? Experiments will be conducted to perform to evaluate whether extended models are ``better'' or not. 

