\section{Motivation}
\label{i:motivation}
In the 21st century, the field of program comprehension research began to focus on applying
various information retrieval (IR) techniques (e.g. text mining, LSI, knowledge-based NL
understanding) to software and its associated text collections (manuals, source, design documents, etc.)  
When this happened, despite known problems with scalability, these exhaustive IR semantic approaches 
became the standard.  With the massive outbreak of the web, the amount of data available on which to do 
computations has grown exponentially.  For this reason, heuristic approaches have acquired a new importance.  

Today, while there have been a few looks into these heuristic algorithms, 
most of the research is still focused on the original exhaustive methods which have
been shown to be NP-complete.  Given that these heuristic methods 
have astounding benefits in terms of execution time due to their heuristic nature,
there has yet to be a comparative study looking at the cost-benefit (or run-time to accuracy)
ratio of heuristic algorithms vs their exhaustive ancestors.

The need for this analysis has arose out of a NARA (National Archives and Records Administration) funded 
project on the future of long-term data retention.  Our toolkit and user interface, Hamlet (see \ref{t:hamlet}), 
was developed primarily for the area of technical document comprehension, specifically the STEP/EXPRESS schema
(see \ref{b:step}).  The goal being to answer the question, "What other designs are similar to this one?".
Because of the massive search space of STEP designs and their associated textual project information 
(documentation, web pages, documentation, etc.), highly scalable algorithms were a must.  Scalability however,
must come at a reasonable price, that is, the loss of accuracy.  Determining the cost of this
scalability is the primary motivation of this research.

\section{Contributions of This Document}

This report has made the following contributions to both the literature and practice, 
with respect to the corpora used in this study:


\begin{itemize}
\item  Determined empirically, the loss of accuracy when using heuristic clustering and
dimensionality reduction algorithms compared to the exhaustive alternatives
\item Determined empirically, the benefits of scalability when using heuristic clustering and
dimensionality reduction algorithms compared to the exhaustive alternatives
\item Determined empirically, the effects of algorithm parameters to the trade-off of
scalability and clustering accuracy when using heuristic clustering and
dimensionality reduction algorithms compared to the exhaustive alternatives
\item Provided recommended algorithms and parameters to optimize the
 scalability/accuracy trade-off in various domains
\end{itemize}


The results of this study have shown that it is possible to greatly reduce the 
cost, in terms of run-time, of the IR methods we have examined with only
a relatively small loss in accuracy.  Different uses of these methods
have varying tolerance for acceptable performance loss.  We have found that in order to
optimize a solution for a specific domain, certain combinations of
clusterers and reduction methods work better than others.  In addition to 
using different combinations of reduction methods and clustering algorithms, 
using different values for the parameters of these algorithms can also have 
drastic effects on both run-time and accuracy that will need to be considered
for the specific application of these methods.  In general, we have found that
the cost of the exhaustive methods doesn't always equal the benefits to accuracy.




\section{Structure of This Document}

The outline of the remaining chapters is as follows:
\begin{itemize}
\item Chapter 2 describes the necessary background information on the techniques used in 
this research along with other standard techniques in the field.
\item Chapter 3 details the tools created as part of this research.
\item Chapter 4 explains the experimental methods used, the datasets the experiment was run on, and 
includes an assessment of the outcomes.
\item Lastly, chapter 5 summarizes the findings of this work and charters the path of future investigations.
\end{itemize}
