\section{Hamlet}
\label{t:hamlet}

\subsection{Motivation}


A recent NSF-funded workshop\footnote{
Collaborative Expedition Workshop \#74, June 10, 2008, at NSF.
``Overcoming I/O Bottlenecks in Full Data Path Processing: Intelligent, Scalable Data Management from Data Ingest to Computation Enabling Access and Discovery''.
\url{http://colab.cim3.net/cgi-bin/wiki.pl?ExpeditionWorkshop/TowardScalableDataManagement_2008_06_10}
} 
 highlighted current directions
in long term technical document retention. While much progress was reported
on:
\bi
\item
systems issues of handling and sharing very large data collection (e.g. SLASH)
\item
scalable methods of building customization views (e.g. iRODS),
\ei
there was little mention of the cognitive issues of 
how users might 
browse and  synthesize
data from massive data collections of technical documents.

For example, here at WVU, we are mid-way through a review of the
use of STEP/EXPRESS for long term technical document retention\footnote{See
reports from Mucino.}.  STEP/EXPRESS is commonly used as an
inter-lingua to transfer technical data between CAD/CAM packages.
Strange to say, while STEP/EXPRESS is useful for transferring and
understanding technical documents {\em today}, it does not appear
to be suitable for understanding technical documents from {\em
yesterday}.



\subsection{Overview}

\input{figures/cycle}

HAMLET's search for association has to two essential phases, and a third optional phase
(see \fig{cycle} ). All these stages have the same goal: from a large set of possible associations,
extract the small subset that (a)~have occurred frequently in prior technical documents and which
(b)~the user will approve. 
 Before running any query:
\bi
\item
 Archival artifacts must be parsed into a network of nodes and edges. 
Each node is represented as a set of {\em terms} referenced in that node.
This set of terms can include every term in the archive so we call these the {\em wide vectors}.
\item
The wide vectors are unwieldy to process. Hence, we run some linear time {\em reduction}
methods to isolate the most informative columns. This produces  the {\em narrow vectors}.
Note that, when we run a query, we only use the parts of the query that appear in these
narrow vectors.
\item
HAMLET also clusters the narrow vectors into {\em groups}. These groups define cliches of repeated
structures.
\ei
When running a query:
\bi
\item We remove irrelevant detail from the query. Linear time methods remove white space 
and spurious word endings. 
Then the query is pruned back to include just the terms seen in the narrow vectors.
\item The resulting {\em tuned vector} is then matched to the the groups of vectors found
above. For computational reasons, we run this match in two-stages. First, we find the N nearest clusters (this generates the {\em relevant groups}). Second, searching just within those relevant groups, we perform exact matches to find related terms.
\item
The {\em candidate matches} found from the exact match are then ranked (by the size of the overlap of the query and the terms) then pruned using thresholding (find the neighboring items in th sort with the biggest difference between them; prune the items below that largest cliff).
\item
The user assesses the matched queries and declares that some are ``relevant'' and some aren't. 
This builds up a session log for this user working these kinds of queries. Once this log grows beyond a certain size, it is used to refine the tuned query such that the tuning favors nodes that
do not contain what the user has labeled ``irrelevancies'' and does contain what the user has called
``relevances''.
\ei
Optionally, we can visualize the results:
\bi
\item The N-dimensions of the vectors are mapped down to 3 dimensions, then visualized on the screen.
\ei

\subsection{Parsing Framework}

HAMLET's language-specific subparsers comb through individual files 
and pull out important bits of information (entities in STEP, methods in Java). 
While processing individual files, these subparsers collect information about each document. 
Some of that information includes pointers to the parsed information and information about what entities are used by others. 
The HAMLET generic parsing framework provides several methods to utilize
these data attributes.

The most important function of the API is the generation of the GraphXML file. 
This file is the intermediary between the data set and HAMLET. It
contains a list of each document (vertice) and the relationships between
them. Other pertinent information, such as file pointers and 
document statistics, is stored in the form of attributes for each 
document vertice. 
From a higher level, the collection of these pointers
to files gives a view of the region of interconnected designs, 
giving HAMLET the ability to make its
decisions and provide suggestions based on what it has already learned.

\begin{figure}[!h]
\begin{center}
\includegraphics[width=4.5in]{figures/graph.png}
\end{center}
\caption{One vertice and the information associated with it.
}\label{fig:vertice}
\end{figure}

For certain language imports, such as Java or STEP, HAMLET utilizes edge
generation to determine the relationship of one design to another. 
For instance, if a call graph is generated
on a set of Java source files, 
an edge can be placed between a multitude of methods and calls made to
and from them. The XML graph generated by the parsing API includes 
both the document vertices and the edges that connect them. 
This is essential for visualization purposes and provides a 
wealth of syntactical information.

Internally, HAMLET makes minimal assumptions about the form of the technical document:
\bi
\item 
A document
contains slots and slots can be atomic or point to other documents;.
\item
The network of pointers between documents presents the space of connected designs.
\ei
A generic parser class implements a standard access protocol for this internal model.
By sub-classing that parser, it is possible to quickly process new documents types.
Currently, HAMLET's parsers can import:
\bi
\item STEP/EXPRESS
\item Florida Law (XML)
\item Text documents structured as follows: sub-headings within headings, paragraphs
within sub-headings, sentences within paragraphs, words in sentences;
\item JAVA:  
This JAVA import allows ready access to very large corpora of
structured technical information (i.e. every open source JAVA program
on the web). Hence, in the sequel, we will make extensive use of
JAVA examples since that permits tests of scalability.
\ei

\subsection{Preprocessor}

HAMLET contains several components other than the UI shown below. One such com-
ponent is the pre-processor, a tool used by HAMLET to generate datasets which can
then be loaded into the UI allowing the user to query, rank, and visualize the documents
found in the loaded dataset. The majority of all machine learning takes place within
the pre-processor. This is where tasks like term frequency / document frequency gen-
eration, term selection, clustering, and learner training occurs. After being run through
the pre-processor, each document within a collection is assigned a vector representation
which describes what terms are present in the document and at what frequency.


\subsection{User Interface}


\begin{figure}[!b]
\begin{center}
\includegraphics[width=3.5in]{figures/whyhamlet.png}
\end{center}
\caption{Looking at data using HAMLET has several visualization advantages
}\label{fig:whyHAMLET}
\end{figure}

\section{Ourmine}
\label{t:ourmine}
\subsection{Motivation}
\subsection{Overview}
\subsection{Textmine}
\label{t:textmine}

\section{The Attribute-Relation File Format (ARFF)}
\label{t:arff}
