\section{Domain Experts}

We investigate techniques for machine learning techniques to act as our domain expert.

\subsection{Motivation}

Compton et al. defines an expert as people whose expertise is scarce and valuable.\cite{COMPTON1995}  A human expert can often be difficult to find.  Humans are also prone to error and are slow when analyzing large amounts of batch data (i.e. software engineering data).  An expert may also possess a bias or may specialize in different strategies than other experts \cite{SHAW1989}.  An active learning scheme may benefit from the input of multiple experts who are capable of processing large amounts of data quickly.    

\subsection{The Use of Simulated Experts in Evaluating Knowledge Acquisition}

A machine learning algorithm may be used to stand in for a domain expert for cases when a human domain expert is unavailable.  The machine learning algorithm is not an ideal substitute of a living, breathing expert, but the simulated expert could improve the knowledge acquisition process.  It is important to note that the expert is only as good as the machine learning algorithm governing it's "opinion" of the data and using only a single expert may not allow the data to be fully explored.  Compton\cite{COMPTON1995} and Shaw\cite{SHAW1989} both suggest the use of multiple experts that can correspond on a problem and "brain-storm" to fully explore their domain of expertise.

\subsection{Comparing Conceptual Structures Between Experts}

Our experts must be able to communicate with each other if they are to discover on what topics they agree or disagree.    Shaw expresses the need for experts to be able to communicate their knowledge of concepts with each other in order to come to a consensus \cite{SHAW1989}.  The author establishes four relationships that can occur between experts:
\bi 
\item \textbf{Consensus: } Experts use the terminology and concepts in the same manner.
\item \textbf{Correspondence: } Experts use different terminology to describe the same concepts.
\item \textbf{Conflict: } Experts use the same terminology to describe different concepts.
\item \textbf{Contrast: } Experts differ in their knowledge of both terminology and concepts.
\ei

\begin{figure}[htb]
\begin{center}
\includegraphics[width=0.8\textwidth]{reperatorygrid.eps}
\end{center}
\caption{An example of a reperatory (entity-attribute) grid \cite{SHAW1989}}
\label{fig:Reperatory}
\end{figure}

We can use a reperatory grid (referred to as an entity-attribute grid in Shaw's literature) as a means of establishing an expert's knowledge on a topic. Figure \ref{fig:Reperatory} shows an example of a reperatory grid.  Experts can exchange these reperatory grids to develop a difference grid (shown in Figure \ref{fig:Difference}) to easily establish consensus and conflict between the experts.  This difference grid populated by scores of 0 (denoted by blank grid locations) to 5 with lower scores indicating a consensus and higher scores indicating conflicts.  Our example shows that our experts agree on concepts in the upper right corner of the grid and show conflict on concepts in the lower left corner of the grid.

\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.8\textwidth]{differencegrid.eps}
\end{center}
\caption{An example of a difference grid \cite{SHAW1989}}
\label{fig:Difference}
\end{figure}

\subsection{ID3 and C4.5}

C4.5 is a decision tree machine learning algorithm developed by Ross Quinlan that is an extension of the ID3 algorithm that addresses several shortcomings of ID3.

\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.8\textwidth]{id3exampletrainingset.eps}
\end{center}
\caption{An example data set to be processed by ID3 \cite{QUINLAN1986}}
\label{fig:ID3ExampleData}
\end{figure}

The ID3 algorithm begins by choosing a subset of the training data set at random.  Quinlan refers to this subset of data as the \textit{window}\cite{QUINLAN1986}.  A decision tree is formed that correctly classifies all objects in the window.  The algorithm attempts to classify the rest of the training set using the generated decision tree.  If the algorithm correctly classifies the rest of the training set, the algorithm terminates; Otherwise, the algorithm adds a selection of incorrectly classified data to the \textit{window} and regenerates the decision tree. \ref{fig:ID3ExampleData} shows an example data set, and Figure \ref{fig:ID3ExampleTree} shows the resulting decision tree produced by ID3.

C4.5 extends ID3 by adding the ability to deal with missing data and numeric data and provides pruning of the decision tree.

\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.8\textwidth]{id3exampledecisiontree.eps}
\end{center}
\caption{An example decision tree produced by the ID3 algorithm \cite{QUINLAN1986}}
\label{fig:ID3ExampleTree}
\end{figure}



%\subsection{Induct}

\subsection{Naive Bayes}

Naive Bayes is a simple statistics-based learning method that is based on Bayes' theorem.  Given a class variable $C$ and a list of features $f_1..f_n)$: 

$p( C | f_1..f_n ) = \frac{p(C)*}{p(f_1..f_n | C)}{ p(f_1..f_n)}$

The Naive Bayes algorithm  is simple to use in practice, very efficient, and robust against noisy data \cite{Yang03}. The algorithm may be used on raw numeric data using a Gaussian probability density function or on discrete data.  A limitation of the Naive Bayes classifier is the assumption of feature independence utilized by the classifier is often violated in the real world.  The Naive Bayes classifier works very well despite this limitation. 

\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.8\textwidth]{bayesisawesome.eps}
\end{center}
\caption{Comparison of Naive Bayes to other algorithms \cite{DOMINGOS97}}
\label{fig:BayesResults}
\end{figure}

Testing by Pedro Domingos has found that Naive Bayes performs better than other classification algorithms including C4.5 \cite{DOMINGOS97}.  Figure \ref{fig:BayesResults} shows Domingos' results comparing Naive Bayes with discretized data (Bayes), Naive Bayes using a Gaussian probability function for the raw numeric data (Gauss), and several other algorithms.  Domingos ranked each classifier using 1 to indicate the most accurate algorithm and increasing as accuracy decreases.  Naives Bayes with discrete data performs best while Naive Bayes using the Gaussian probability density function ranked second best.

