\section{Introduction}

OURMINE is a scripting environment developed at West Virginia University for the development and deployment of data mining experiments. 
Using this toolkit, those new to data mining can specify and execute complex experiments, while researchers in the field can publish their entire experimental rig alongside their results. Sharing experiments is important for not only the reproduction of results, but also in {\em understanding} those results -- by examining the experiment itself, more insight can be gained from theories involved in the experiment's execution. This is achievable due to OURMINE's succinctness. For example, this thesis presents four case studies documented in the OURMINE syntax. Thus, the brevity and simplicity of OURMINE recommends it as an ideal tool for not only constructing experiments, but also documenting, executing, and sharing them with the data mining community. 

 The toolkit primarily uses UNIX shell scripting and GAWK~\cite{awkbook}. As a result, any tool that can be executed from a command prompt can be easily and seemlessly combined with other tools. As I will show in this thesis, components written in various languages such as Perl, Java, GAWK, BASH, etc. can be connected through the environment of OURMINE. Therefore, this strictly scripting approach can be thought to provide the ``glue'' needed to create an interconnected network of components required in performing data mining experiments.

Also demonstrated in this thesis are the advantages of using scripting languages for not only research, but also in teaching data mining concepts to university students. In essence, while learning complex tasks in a visual environment is more than possible, the user is left with only knowledge of how to use properties specific to that environment. On the other hand, learning to use modern and popular scripting languages utilized by a toolkit such as OURMINE leaves the user with knowledge that can be brought to many general applications outside of data mining. 

%Since 2004, data mining researchers at West Virginia University have been involved in an open source data mining experiment. Researchers
%submitting to the PROMISE conference on predictive models in software engineering are encouraged to offer not just conclusions,
%but also the data they used to generate those conclusions. The result is nearly 100 data sets, all on-line,
%freely available for download~\footnote{\url{http://promisedata.org/data}}. 

%An important aspect of this thesis is that after {\em data sharing} comes {\em experiment
%sharing}; i.e. repositories should grow to include not just data, but the code
%required to run experiments over that data. This simplifies the goal of letting other researchers 
%repeat, or even extend, data mining experiments of others. Repeating experiments is not only essential 
%in confirming previous results, but also in providing an insightful understanding of the fundamentals 
%involving those experiments. Thus, it is important to have a prepared set of clearly defined and operational experiments that can be utilized in order to achieve similar prior results.     

%OURMINE was developed to help graduate students at West Virginia University document and share their 
%data mining experiments.
%The toolkit 
%uses UNIX shell scripting. As a result, any tool
%that can be executed from a command prompt can be seamlessly combined with other tools.

%For example, \fig{clean} shows a simple bash
%function used in OURMINE to clean text data before conducting any
%experiments using it. Line 5 passes text from a file, performing
%tokenization, removing capitals and unimportant words found in a
%stop list, and then in the next line performing Porter's stemming
%algorithm on the result. 

%OURMINE allows connectivity between tools
%written in various languages as long as there is always a command-line
%API available for each tool.
%For example, the modules of \fig{clean} are written using BASH,
%Awk and Perl.  

%\begin{figure}
%{\scriptsize\begin{verbatim}
% 1 clean(){
% 2    local docdir=$1
% 3    local out=$2
% 
% 4   for file in $docdir/*; do
% 5      cat $file | tokes | caps | stops $Lists/stops.txt > tmp 
% 6	     stems tmp >> $out 
% 7	     rm tmp
% 8   done
% 9 }\end{verbatim}}
%\caption{An OURMINE function to clean text documents and collect the results. 
%$Tokes$ is a tokenizer; $caps$ sends all words to lower case;
%$stops$ removes the stop works listed in "\$Lists/stops.txt"; and
%$stems$ performs Porter's stemming algorithm (removes confusing suffixes).
% }\label{fig:clean}
%\end{figure}

The following sections describe OURMINE's functions and applications.

\section{Built-in Data and Functions}

In order to encourage more experimentation, 
the default OURMINE installation comes with numerous data sets:

\bi
\item {\em Text mining data sets:} including STEP data sets (numeric): ap203, ap214, bbc, bbcsport, law, 20 Newsgroup subsets [sb-3-2, sb-8-2, ss-3-2, sl-8-2]\footnote{\url{http://mlg.ucd.ie/datasets}}
\item {\em Discrete UCI Machine Learning Repository datasets:} anneal, colic, hepatitis, kr-vs-kp, mushroom, sick, waveform-5000, audiology, credit-a, glass, hypothyroid, labor, 
pcolic, sonar, vehicle, weather, autos, credit-g, heart-c, ionosphere, letter, primary-tumor, soybean, vote, weather.nominal,breast-cancer, diabetes, heart-h, iris, lymph, segment, splice, vowel;
\item {\em  Numeric UCI Machine Learning Repository datasets:} auto93, baskball, cholesterol, detroit, fruitfly, longley, pbc, quake, sleep, autoHorse, bodyfat, cleveland, 
echoMonths, gascons, lowbwt, pharynx, schlvote, strike, autoMpg, bolts, cloud, elusage, housing, mbagrade, pollution, sensory, veteran, 
autoPrice, breastTumor, cpu, fishcatch, hungarian, meta, pwLinear, servo, vineyard.
\item The defect prediction data sets from the {\em PROMISE repository}:  CM1, KC1, KC2, KC3, MC2, MW1, PC1
\ei

OURMINE also comes with a variety of built-in functions to perform data mining and text mining tasks. For a complete list, see the appendix.

%For a sample of these functions, see \fig{functions}. 
%\begin{figure}
%\small
%\begin{description}
%\item[abcd] provides analysis of experiments, such as {\em pd,pf,accuracy} and {\em precision} values;
%footnotetext[1]{
%If $\{a,b,c,d\}$ are the
%true negatives, 
%false negatives, 
%false positives, and 
%true positives
%(respectively) found by a binary detector then 
%$pd=recall=\frac{d}{b+d}$,
%$pf=\frac{c}{a+c}$,
%$prec=precision=\frac{d}{d+c}$,
%$acc=accuracy=\frac{a+d}{a+b+c+d}$,
%$f=\frac{2*pd*pf}{pd + pf}$,
%and 
%$balance=1-\sqrt{(1-pf)^2 + pd^2}/\sqrt{2}$ .}
%\item[clean] clean text for further processing, removing tokens, capitalizations, stop words, etc.;
%\item[docsToSparff] constructs a sparse arff file based on a directory of documents;
%\item[docsToTfidfSparff] generates a sparse arff file of TF-IDF values based on a directory of documents;
%\item[funs] shows a sorted list of all available functions in OURMINE;
%\item[logArff] logs all numeric values in a data set ;
%\item[malign] neatly aligns text into columns;
%\item[nb] Runs Naive Bayes on the data given;
%\item[rankViaInfoGain] ranks attributes by InfoGain values;
%\item[makeTrainAndTest] splits a dataset into a test set and a training set as $train$.arff and $test$.arff, as well as $train$.lisp and $test$.lisp. 
%\end{description}
%\caption{A small sample of the available OURMINE functions. Built-in functions give the user something with which to start and begin running demos and experiments immediately. For a detailed list of available tools, please see the appendix.}\label{fig:functions}
%\end{figure}


\begin{figure}
{\scriptsize\begin{verbatim}
function train() { #update counters for all words in the record 
  Docs++; 
  for(I=1;I<NF;I++) { 
    if( ++In[$I,Docs]==1) 
    Doc[$I]++ 
    Word[$I]++ 
    Words++ } 
} 
function tfidf(i) { #compute tfidf for one word 
  return Word[i]/Words*log(Docs/Doc[i]) 
} \end{verbatim}}
\caption{A GAWK implementation of TF-IDF.}\label{fig:tfidf}
\end{figure}

%\subsection{Adding/Modifying OURMINE Code}

%OURMINE uses an interpreted system, thus, it is a simple matter to extend or modify its functionality.
%For example, if a user wished to add a {\em myFunctions.sh} to be used in the environment

%\begin{itemize}
%\item{Create an executable {\em myFunctions.sh} containing any custom scripts}
%\item{Include this file in {\em \$HOME/opt/ ourmine/our/lib/sh/minerc.sh }}
%\end{itemize}

%Once this basic structure of OURMINE is learned, however, more advanced additions can be made such as:
%including multiple directories of BASH files, some of whose functions call code written in other files and other languages located elsewhere in the codebase. This gives a nearly limitless opportunity for customization.

%\section{Extending Ourmine Code}  
%\section{Writing Experiments Using Ourmine}
%   \subsection{Automating \& Parallelizing Experiments}
%\section{Analyzing Experimental Results}
\section{Learning \& Teaching with Ourmine}

Data mining concepts become complex when implemented in a complex manner.
For this reason, OURMINE utilizes
simple scripting tools (written mostly in BASH or GAWK) to
better convey the inner-workings of these concepts. For instance,
\fig{tfidf} shows a GAWK implementation used by OURMINE to determine
the TF-IDF~\cite{ramos03} (Term Frequency * Inverse Document Frequency, described below) values of each term in a document. This
script is simple and concise, while a C++ or Java implementation
would be large and overly complex. An additional example demonstrating the brevity 
of OURMINE script can be seen in \fig{demo004}, which is a complete experiment whose  
form can easily be taught and duplicated in future experiments.

\begin{figure}
{\scriptsize\begin{verbatim}
1 demo004(){
2     local out=$Save/demo004-results.csv
3      [ -f $out ] && echo "Caution: File exists!" || demo004worker $out
4 }

5 # run learners and perform analysis
6 demo004worker(){

7     local learners="nb j48"
8     local data="$Data/discrete/iris.arff"
9     local bins=10
10    local runs=5
11    local out=$1
    
12    cd $Tmp
13    (echo "#data,run,bin,learner,goal,a,b,c,d,acc,pd,pf,prec,bal"
14	   for((run=1;run<=$runs;run++)); do
15	       for dat in $data; do
		
16		          blab "data=`basename $dat`,run=$run" 
17		          for((bin=1;bin<=$bins;bin++)); do
		    
18		              rm -rf test.lisp test.arff train.lisp train.arff
19		              makeTrainAndTest $dat $bin $bin 
20		              goals=`cat $dat | getClasses --brief`
		    
21		              for learner in $learners; do
			
22			                  $learner train.arff test.arff | gotwant > produced.dat
23			                  for goal in $goals; do
			    
24			                  cat produced.dat | 
25			                  abcd --prefix "`basename $dat`,$run,$bin,$learner,$goal" \
26				                      --goal "$goal" \
27				                      --decimals 1
28			                  done
29		              done
30		          done
31		      blabln
32	      done
33	  done | sort -t, -r -n -k 11,11) | malign  > $out
    
34   winLossTie --input $out --test w --fields 14 --key 4 --perform 11 
35  }
\end{verbatim}}
\caption{A demo OURMINE experiment. This worker function begins by being called by the top level function {\em demo004} on lines 1-4. Noteworthy sections of the demo code are at: line 19, where training sets and test sets are built from 90\% and 10\% of the data respectively, lines 25-27 in which values such as {\em pd,pf} and {\em balance} are computed via the {\em abcd} function that computes values from the confusion matrix, and line 34 in which a {\em Wilcoxon} test is performed on each learner in the experiment using {\em pd} as the performance measure.}
\label{fig:demo004}
\end{figure}


\begin{figure}
{\scriptsize\begin{verbatim}
#naive bayes classifier in gawk
#usage:  gawk -F, -f nbc.awk Pass=1 train.csv Pass=2 test.csv

Pass==1 {train()}
Pass==2 {print $NF "|" classify()}

function train(    i,h) { 
   Total++;
   h=$NF;     # the hypotheis is in the last column
   H[h]++;    # remember how often we have seen "h"
   for(i=1;i<=NF;i++) {
     if ($i=="?") 
           continue;         # skip unknown values
     Freq[h,i,$i]++
     if (++Seen[i,$i]==1) 
           Attr[i]++}  # remember unique values
}
function classify(         i,temp,what,like,h) {  
   like = -100000;         # smaller than any log
   for(h in H) {           # for every hypothesis, do...  
     temp=log(H[h]/Total); # logs stop numeric errors
     for(i=1;i<NF;i++) {  
       if ( $i=="?" ) 
             continue;     # skip unknwon values
       temp += log((Freq[h,i,$i]+1)/(H[h]+Attr[NF])) }
     if ( temp >= like ) { #  better hypothesis
        like = temp
        what=h}
   }
   return what;
}\end{verbatim}}
\caption{A\nbc classifier for a CSV file, where the class label is found in the last column.}\label{fig:nbc}
\end{figure}

\newpage

%\begin{figure}
%\small
%\be
%\item Modify \fig{nbc} so that there is no train/test data. Instead, make it an incremental learning.
%Hint: 1) call the functions $train$, then $classify$ on every line of input. 2) The order is important: always
%$train$ before $classifying$ so the the results are always on unseen data.
%\item Convert \fig{nbc} into HYPERPIPES~\cite{Eisenstein04}. Hint: 1) add globals $Max[h,i]$ and $Min[h,i]$ to 
%keep the max/min values seen in every column ``$i$''and every hypothesis class ``$h$''. 2) 
%Test instance belongs to the class that most overlaps the attributes in the test instance.
%So, for all attributes in the test set, sum the returned values from $contains1$:
%{\scriptsize\begin{verbatim}
%function contains1(h,i,val,numerip) {
%  if(numericp)
%      return Max[h,i] >= val && Min[h,i] <= val
%  else return (h,i,value) in Seen
%  }\end{verbatim}}
%\item Use \fig{nbc} to implement an anomaly detector. Hint:  1)~make all training examples get the same class;
%2) an anomalous test instance has a likelihood $\frac{1}{\alpha}$ of the mean likelihood
%seen during training ($alpha$ needs tuning but $alpha=50$ is often useful).
%\item Using your solution to \#1, create an incremental version of HYPERPIPES and an anomaly detector.
%\ee
%\caption{Four Introductory OURMINE programming exercises.}\label{fig:ourmine101}
%\end{figure}



%\begin{figure}
%\begin{center}
%\includegraphics[width=5.5in]{edits2.ps}
%\end{center}
%\caption{Editing ourmine}\label{fig:editscreen}
%\end{figure}


Another reason to prefer scripting in OURMINE over the complexity of 
\mbox{RAPID-I}, WEKA, ``R'', etc,
is that it reveals
the inherent
simplicity of many of our data mining methods. For example, \fig{nbc} shows a GAWK implementation of
a\nbc classifier for discrete data where the last column stores the class symbol.
This tiny script  is no mere toy- it successfully executes on very large
data sets such as those seen in the 2001 KDD cup and in~\cite{orrego04}. WEKA cannot process these large
data sets since it always loads its data into RAM. \fig{nbc}, on the other hand, only requires
memory enough to store one instance as well as the frequency counts in the hash table ``$F$''.

More importantly, in terms of teaching, \fig{nbc} is easily customizable. 
%\fig{ourmine101} shows four warm-up exercises for novice data miners that (a)~introduce them
%to basic data mining concepts and (b)~show them how easy it is to script their own data miner:
%Each of these tasks requires changes to less than 10 lines from \fig{nbc}. 
The simplicity of this customizations fosters a spirit
of ``this is easy'' for novice 
data miners. This in turn empowers them to design their own extensive and elaborate experiments. 


Also from the teaching perspective, demonstrating on-the-fly a particular data mining concept helps not only to solidify this concept, but also gets the student accustomed to using OURMINE as a tool in a data mining course. As an example, if a\nbc classifier is introduced as a topic in the class, an instructor can show the workings of the classifier by hand, and then immediately afterwards complement this by running\nbc on a small data set in OURMINE. Also, since most of OURMINE does not use pre-compiled code, an instructor can make live changes to the scripts and quickly show the results.%, as in \fig{editscreen}. 

%\begin{figure}
%\begin{center}
%\includegraphics[width=4in]{edits}
%\end{center}
%\caption{The image above depicts running $demo003$ in OURMINE, which then calls $demo002$. The first 
%execution in the top window demonstrates running the J48 decision tree classifier on the UCI $iris$ data set. Then, in the bottom window, the data set is changed to $weather$. Once any modification is made to the code of OURMINE, the environment can be reloaded without exiting. This is done by entering $reload$, as seen in the above window. Finally, the demo is executed again, yielding a different result.
%}
%\label{fig:editscreen}
%\end{figure}

Data mining colleagues at WVU are not alone in favoring GAWK for teaching purposes.
Ronald Loui uses GAWK to teaching artificial intelligence
at Washington University in St. Louis. He writes:

\begin{quote}{\em
There is no issue of user-interface. 
This forces the programmer to return to the question of what the program does, not how it looks. 
There is no time spent programming a binsort when the data can be shipped to /bin/sort in no time.} ~\cite{gawkai}
\end{quote}

%Another reason for OURMINE's scripting approach is due to its applicable generality. If a student learns a highly specific portion of a toolkit, those learned skills are generally confined to that environment. However, skills obtained from learning to use scripting within OURMINE can be used in a variety of future applications.

\begin{figure}
{\scriptsize\begin{verbatim}
Function: j4810
Arguments: <data (arff)>
Example(s): j4810 weather.arff
Description: Uses a j48 decision tree learner on the input data

Function Code:
==============
j4810 () { 
    local learner=weka.classifiers.trees.J48
    $Weka $learner -C 0.25 -M 2 -i -t $1
} \end{verbatim}}
\caption{Function help in OURMINE.}\label{fig:help}
\end{figure}
 

Function documentation provides a way for newcomers to OURMINE to not only get to know the workings of each function, but also add to and modify the current documentation. Instead of asking the user to implement a more complicated ``man page'', OURMINE uses a very simple system consisting of keywords such as {\em name, args} and {\em eg} to represent a function name, its arguments and an example of how to use it. Using this documentation is simple. Entering {\em funs} at the OURMINE prompt provides a sorted list of all available functions in ourmine. Then, by typing {\em help X}, where {\em X} is the name of the function, information about that function is printed to the screen. See \fig{help} for an example of viewing the help document for the function {\em j4810}. Documentation for a function is added by supplying a text file to the helpdocs directory in OURMINE named after the function.

%From the teaching perspective, demonstrating on-the-fly a particular data mining concept helps not only to solidify this concept, but also gets the student accustomed to using OURMINE as a tool in the course. 
%\fig{demo004} shows the canonical OURMINE example. XXX more details.
%Once a student learns the basic mechanics of this demo experiment, its framework can be copied to another function within OURMINE to conduct a custom experiment using any number of tools.

\section{Using Ourmine}

While this section represents important and helpful "tips and tricks" for immediately getting up and running with OURMINE, it also serves a purpose as a stand-alone manual to be read by students taking an introductory data mining course to quickly familiarize themselves with the environment. 
	Therefore, subsequent sections are broken into the following:
				
	\bi	
		\item{\bf An understanding of the OURMINE environment}				
	\ei	
	
		In order to use the environment quickly and effectively, having a basic awareness of its inner-workings is essential. Thus, I first give a brief but ample overview of how OURMINE works by utilizing neatly distributed code segments. This alone should give way to the ability to add and modify scripts as desired.
	
	\bi
		\item{\bf How to construct a simple experiment in OURMINE using easy scripting}
	\ei	
	
		Building experiments in OURMINE is simple. However, there are certain conventions that must be understood and adhered to before any true experimentation can be conducted using it. This portion discusses how to, in detail, construct a simple data mining experiment using OURMINE. The results from this experiment are used in further sections to demonstrate analyses.
		
	\bi
		\item{\bf Ranking experimental treatments using statistical tests}
	\ei
	
		Comparing experimental results is as important as building the experiment itself. For this reason, two approaches to analyze experimental results will be discussed here. First, I show how to rank treatments from an experiment using OURMINE's built-in statistical ranking tests.
		
	\bi
		\item{\bf Examining the variance and median values of experimental results using quartile charts}	
	\ei
	
		Second, I show how to analyze the variance and median values after an experiment has executed. This is essential, in combination with the statistical ranking, to measure the overall performance of a treatment used in any data mining experiment.
	
\subsection{The Environment}
	
	OURMINE's environment was constructed with {\em modularity} in mind. Each and every different type of process used in the toolkit is located in its own location and give way to easily customizable scripts. Here I will begin from the initial execution of the environment to modifying scripts to provide a better understanding of how the interconnected code of OURMINE works as a whole. By the time this section is finished, the user should have a working knowledge of where code segments are located in OURMINE, and also how to modify and add new segments as desired.
	After installing OURMINE (see the Appendix), the default prompt is given, as in \fig{ourmine-homescreen}. 
	
	
		
	\begin{figure}
		\begin{center}
		\includegraphics[width=6in]{figures/ourmine-firstprompt.png}
		\end{center}
		\caption{The OURMINE homescreen after installation.}\label{fig:ourmine-homescreen}
    \end{figure}
    
   	This prompt notifies the user that the OURMINE environment is available for use, regardless of location. Since the data mining environment is loaded into memory in {\em addition} to standard shell commands (this becomes very powerful in later use), normal command line operations can still be utilized to its full extent while in OURMINE. To exit the environment, type {\bf exit} at the prompt. To re-enter OURMINE, navigate to where the toolkit was installed, for example in the default location {\em \$HOME/opt/ourmine/our}, and issue the command {\bf ./ourmine}. By executing the script $ourmine$, a variable is set to the current directory (default install), and then this value is passed to a shell script called $minerc.sh$, located in {\em \$HOME/opt/ourmine/our/lib/sh}. While this may sound complicated, it's actually very simple. \fig{ourmine-minerc} shows the contents of the $minerc$ file as it is upon installation. Note that this can grow to become much more large and complex as code is added to the environment as desired. 
   	 
	
	
	\begin{figure}
		\begin{center}
		{\scriptsize
		\begin{verbatim}
		#define and create required directories

Base=$OurMine
Data=$Base/arffs
Docs=$Base/docs
Help=$Base/helpdocs
Tmp=$HOME/tmp
Var=$Tmp/var
Awk=$Base/lib/awk
Sh=$Base/lib/sh
Java=$Base/lib/java
Perl=$Base/lib/perl
Lists=$Base/lib/lists
Save=$Base/save

mkdir -p "$Var $Tmp"
mkdir -p $Tmp

# useful globals

Weka="java -Xmx2048M -cp $Java/weka.jar "
Clusterers="java -Xmx1024M -jar $Java/Clusterers.jar "
Reducers="java -Xmx1024M -jar $Java/Reduce.jar "

# define and load files

Files=" 
                $Sh/effort.sh
                $Sh/util.sh 
                $Sh/preprocess.sh 
                $Sh/learn.sh 
                $Sh/cluster.sh
                $Sh/fss.sh
                $Sh/analysis.sh
                $Base/workers/worker_cluster.sh
                $Base/workers/worker_cluster_analysis.sh
                $Base/workers/worker_reduce.sh
                $Base/workers/worker_defects.sh
                $Base/workers/worker_learner_analysis.sh
                $Base/demos.sh          
                $Base/demos1.sh         
                $Base/demos2.sh         
                $Base/demos3.sh         
                $Base/demos4.sh         
                $Base/demos5.sh         
                $Base/demos6.sh         
                $Base/demos7.sh         
                $Base/demos8.sh         
                $Base/demos9.sh         
                $Base/demos10.sh                
                "
#load all from Files above

for config in $Files; do 
        [ -f  "$config" ] && . $config
done

echo "Ourmine - Copyright 2009 by Tim Menzies, Adam Nelson, Gregory Gay"
PS1="OURMINE> "
		\end{verbatim}}
		\end{center}
		\caption{The $minerc$ file used in OURMINE for setting up the environment.}\label{fig:ourmine-minerc}
    \end{figure}

	As can be seen in \fig{ourmine-minerc}, the variable {\bf Base} is set to the default installation path. Using this variable, we can construct other variables representing locations of important directories within OURMINE. For example, the variable {\bf Data} is set to the location of the {\em \$HOME/opt/ourmine/our/arffs} directory. This is where all of the data that comes packaged with OURMINE is stored. On the same note, the variables {\bf Sh} and {\bf Java} store the locations of shell code to be executed, as well as java code respectively.
	
	These variables set to their respective paths can be used for many applications. For instance, we can see in \fig{ourmine-minerc} that {\bf \$Java} is used to set another variable, {\bf \$Weka}, which allows us to now access all of the data mining properties of the Weka toolkit. Afterwards, every file declared in the string {\bf \$Files} is used to import all BASH functions from their respective files. Let's examine further one of these files, such as {\em \$Sh/learn.sh}. 
	
	{\em learn.sh} is one of many shell scripting files located in {\em \$HOME/opt/ourmine/our/lib/sh}, and each are named according to use; {\em util.sh} contains code for standard utility functions used in building experiments in OURMINE, {\em fss.sh} utilizes code crucial for Feature Subset Selection (FSS), {\em cluster.sh} has scripts responsible for calling code used to cluster data, etc. Since our main concern is with {\em learn.sh} at the moment, I will spend a bit of time discussing how this works, and how its scripts can easily be modified and extended for further use.
	
	\fig{ourmine-learning} shows a subset of the available functions in {\em learn.sh} that can be accessed from the OURMINE prompt. Here, we can see the function {\bf nb()} that is used to call Weka's\nbc using as input a training set, followed by a testing set. Likewise, {\bf j48()} executes the J48 decision tree on the training set. Notice those functions ending in 10, such as {\bf nb10} and {\bf j4810}. This is an arbitrary convention that has been used since the earlier days of OURMINE's construction, and simply means that there is no testing data provided, and thus all testing is conducted on the training set.  
	
		\begin{figure}
		\begin{center}
		{\small
		\begin{verbatim}
nb() {
        local learner=weka.classifiers.bayes.NaiveBayes
        $Weka $learner -p 0 -t $1 -T $2  
}
nb10() {
        local learner=weka.classifiers.bayes.NaiveBayes
        $Weka $learner -i -t $1   
}
j48() {
        local learner=weka.classifiers.trees.J48
        $Weka $learner -p 0 -C 0.25 -M 2 -t $1 -T $2
}
j4810() {
        local learner=weka.classifiers.trees.J48
        $Weka $learner  -C 0.25 -M 2 -i -t $1 
}
zeror() {
        local learner=weka.classifiers.rules.ZeroR
        $Weka $learner -p 0 -t $1 -T $2
}
zeror10() {
        local learner=weka.classifiers.rules.ZeroR
        $Weka $learner -i -t $1
}
		\end{verbatim}}
		\end{center}
		\caption{A few of the contents of the $learn.sh$ file used in OURMINE for data training and classification.}\label{fig:ourmine-learning}
    \end{figure}
    
    Running a learner on input data is simple, and is detailed in \fig{ourmine-j48}. At the top of the screen shot, we can see the command {\bf j4810 \$Data/discrete/weather.arff} being issued to OURMINE. Internally, this calls Weka's J48 decision tree classifier on the extremely popular weather data set found in the {\bf \$Data} directory under discrete (for discrete data). Further information about the results of running the learner is also printed to the screen, such as the decision tree itself (and corresponding information pertaining to it), the execution time, classification accuracy, and per-class statistics obtained from the confusion matrix, such as precision and recall. 
    
    \begin{figure}
		\begin{center}
		\includegraphics[width=6in]{figures/ourmine-j48.png}
		\end{center}
		\caption{Running the J48 algorithm in OURMINE.}\label{fig:ourmine-j48}
    \end{figure}
    
    Configuring shell functions in OURMINE for any desired task is simple. For example, suppose a user wished to import the OneR rule learner to be used in future experiments. This operation requires the addition of one BASH function. The example in \fig{ourmine-add-oner} shows that the environment does not recognize the command {\bf oner10} at first using the input training data. However, by simply editing {\em learn.sh} to include the function code for the learner, and then supplying the command {\bf reload} (discussed in Section \ref{tandt}), the environment is {\em reloaded} and now contains working function code. \fig{ourmine-add-oner} shows the rules learned from running the newly created function.
    
    Thus, in summary, OURMINE's structure operates by taking advantage of BASH functions located in clearly defined files. These functions, however, call any number of other functions and libraries written in almost any other programming language. The result is an extremely powerful "sandbox" in which massive amounts of modification and experimentation can take place.
    
    \begin{figure}
		\begin{center}
		\includegraphics[width=7.0in, height=5.0in]{figures/ourmine-adding-oner.png}
		\end{center}
		\caption{Adding Weka's One-R implementation into OURMINE for further use.}\label{fig:ourmine-add-oner}
    \end{figure}
			 
\subsection{Tips, Tricks and Useful Ourmine Functions}\label{tandt}

This section highlights the more helpful and abundantly used functions in OURMINE that, when used effectively, drastically increase the speed and ease at which the environment can be used for data preparation, experimentation, analysis and more.

The first, and perhaps most helpful, function to examine is {\bf funs}. {\bf funs} is a function requiring no parameters that returns a sorted list of the available functions in OURMINE. This is a useful utility ready for use from the command line that enables a user to identify a function by its use through searching for keywords. For example, suppose the user knew that a function existed that could be used to align data, but cannot recall its name. Instead of flipping through a lengthy manual, the command line itself can be used extremely quickly. By entering {\bf funs} from the OURMINE prompt, a large list of functions can be seen. Since an alignment function is in question, he/she can provide the BASH utility {\em grep} with the output from the {\bf funs} command, as in 

\begin{verbatim}
	OURMINE> funs | grep align
	malign
	OURMINE> 
\end{verbatim}

From this it is immediately obvious that {\bf malign} is the function desired. 

Another very important utility function within OURMINE is {\bf show}. {\bf show} provides the user with the contents of the function in question. This drastically assists in the identification of the number, order and types of parameters to pass to the function. For instance, \fig{ourmine-show-random} shows the code and parameters for the function {\bf selectRandomInstances}, which simply selects instances at random up to a number that is user-specified. There we can see that in order to successfully call the function, the path of the input file (in arff format), and the number of instances to select. The result is printed to the terminal where further actions can be performed on the output.

\begin{figure}
		\begin{center}
		\includegraphics[width=6in]{figures/ourmine-show-random.png}
		\end{center}
		\caption{Using {\em show} to examine the code (and hence, parameters) of the built-in function {\bf selectRandomInstances}.}\label{fig:ourmine-show-random}
\end{figure}

Modifying or adding new shell scripts to OURMINE is done easily with the use of the {\bf reload} command. {\bf reload} simply reloads the scripts of all supplied OURMINE functions into working memory (see Section \fig{ourmine-minerc}). This way, as in \fig{ourmine-add-oner}, custom scripts can be added to or modified within the environment without having to tirelessly load the environment. For example, two terminals can be used simultaneously for editing and reloading/execution. This encourages rapid modification and experimentation to the existing code-base of OURMINE.

Printing to the screen to provide updates during an experiment's execution can be essential; particular segments of the experiment can be identified as problematic if they yield a large runtime. In most cases, the standard {\em echo} command issued through the terminal is not sufficient for this purpose. To illustrate, the demo ourmine experiment of \fig{demo004} encapsulates the output of the experiment through the use of {\em echo} on lines 13 and 33. This is very useful, as this output data can be neatly filtered and aligned as desired by the experimenter. Line 16 prints to the screen the current data set of the experiment, as well as the run number. Using {\em echo} for this would not have a desirable effect, however, as it would incorporate this text into the output file. Consequently, the results from the experiment would contain extraneous, unwanted information, as well as making the experiment progress of Line 16 not visible during execution. Thus, OURMINE uses the {\bf blab} and {\bf blabln} commands, which print to the screen without or with (respectively) a newline character, using forked processes. This allows the periodic output of experiments to be shown without being combined with the results of the experiment. 

Missing values in training data can yield problems for many classifiers, and thus replacing them may be used as a preprocessing step to prepare data for learning. For this reason, OURMINE uses the {\bf replaceMissingValues} function. This function calls Weka's Java code in order to replace missing values, labeled as "?", by the mode for nominal attributes, and the mean for numeric attributes. The function is used by supplying the name ({\bf replaceMissingValues}), with the location of the training arff as a parameter.

Raw results from experiments can not only be harsh on the eye, but can also complicate the analysis of them if not properly aligned. To combat this, OURMINE uses the {\bf malign} function. When output, in a comma separated form, is passed to {\bf malign}, columns are neatly spaced to allow for easier reading. \fig{ourmine-malign} shows this command in action. A sample of the output from the experiment in \fig{demo004} is shown in the upper portion of the terminal {\em without} using {\bf malign} (the use of this function was removed from line 33 of \fig{demo004} for this example). The bottom half of the terminal demonstrates the function being used by reading the file first using {\bf cat}, and then piping the output to the function. The result is a much neater and easily readable format.

\begin{figure}
		\begin{center}
		\includegraphics[width=7in]{figures/ourmine-malign.png}
		\end{center}
		\caption{Using {\em malign} to neatly align comma-separate output.}\label{fig:ourmine-malign}
\end{figure}


Learners are assessed by how well they perform on unseen data. This is important for real world applications, as the classifiers are always used to make predictions about the future -- data that has not yet been obtained. To test this performance, the standard method is to break up the original data set into a training set and a testing set. The training set is used by the classifier to learn intrinsic characteristics about the data, and the testing set is used to make predictions (hopefully correct) based on these learned characteristics. 

Breaking up the data set is simple in OURMINE, and is conducted by calling the function {\bf makeTrainAndTest}. {\bf makeTrainAndTest} requires the user to specify 1.) the original data set, 2.) the total number of bins to use and 3.) the bin to select as the testing set. For a typical 10-way cross validation, the total number of bins to select will be 10, while the bin to choose for the random testing set is chosen by the user. Using the function is as follows

\begin{verbatim}
OURMINE> makeTrainAndTest $Data/discrete/weather.arff 10 2
\end{verbatim}

places {\em train.arff}, {\em test.arff} as well as train.lisp and test.lisp in the current directory. For instance, if the above was executed in the \$HOME directory, the resulting files would be placed in that directory. From this point, a classifier, such as\nbc, can be trained on {\em train.arff} and tested using {\em test.arff} by calling

\begin{verbatim}
OURMINE> nb train.arff test.arff
\end{verbatim}

Quartile charts provide a succinct representation of experimental results, and therefore provide the ability to quickly assess the performance of treatments. This is due to the ability of the quartile chart to represent minimum, maximum and medians pertaining to any combinations of methods used in the experiment. OURMINE provides the ability to create ASCII quartile charts by calling {\bf quartile}. {\bf quartile} operates on columns of data supplied to it through any filtering means desired by the user. For example, assume an output CSV file (the construction of which is explained in Section \ref{building-exp}) from an experiment contains statistics about the performance of training a\nbc classifier, among others. Further assume that the metric in question is, say, probability of detection ($pd$) and is located in column number 12 (from left to right). Quartiles can be obtained from this file by extracting the column first, and supplying the output to the {\bf quartile} function. For example, the following would provide statistics about the pd for {\em every} classifier or treatment used in the experiment.  

\begin{verbatim}
OURMINE> cat results.csv | cut -f12 -d, | quartile
 10.0, 31.0, 54.0, 77.0,100.0,[    -----------           |          ++++++++++++ ]
\end{verbatim}

The quartile chart shown works as follows. The minimum and maximum values (in this case for pd) are the floating point numbers found at the very left and just before where the ASCII chart begins, or 10.0 and 100.0 respectively. The median value is shown in the center, as 54.0, and the 25\% and 75\% quartiles are shown as 31.0 and 77.0 respectively. The ASCII representation provides a quick visualization of the distributions between quartile values. For instance, the dashes, starting at 10.0 and moving to 31.0 display the variance between the minimum value and the first quartile. The space between the dashes and the upright "pipe", |, represent the variance between the first quartile and the median. The median, therefore, is depicted by the pipe. The same holds true for the markings above the median; the space between the pipe and the first "+" represents the the variance between the median and the third quartile.

It should be noted, however, that the above does not concentrate on one particular classifier, in this example. To obtain quartiles, for, say, only\nbc, we can extract quartiles that meet a particular filter, such as "nb" (short for \nbc).

\begin{verbatim}
OURMINE> cat results.csv | grep nb | cut -f12 -d, | quartile
 42.0, 55.0, 70.0, 85.0,100.0,[                    -------       |      ++++++++ ]
\end{verbatim}
  
Here we can intuitively see a smaller variance in the statistics, as indicated by the decreased length in the horizontal dashed and "+" lines, as well as the existence of fewer spaces between each line and the median.


\subsection{Building a Simple Experiment}\label{building-exp}

The ability to build an experiment using OURMINE is quite possibly the most important skill required when using the toolkit to its full potential. For example, the experiments in Chapters \ref{cs-retention} to \ref{cs-component} detail results obtained from experiments constructed in OURMINE. 

As this is such a crucial skill when working within the environment, I have written here a tutorial on how simple experiments are constructed using the shell scripting of OURMINE. While the following is a smaller scale experiment, the exercises used here can easily be extended to a much larger and more advanced experiment once the user knows both shell scripting and also the details of how the toolkit functions. It is hoped that this section will be a satisfactory introduction to building experiments so that the user of OURMINE can utilize the skills learned here and apply them to his/her own unique data mining experiments.

For this section I am going to start from scratch to build a complete experiment. Then, in Section \ref{evaling}, I show how the results obtained from the work in this section can be analyzed. Thus, the end result will be a fully operational experiment whose results could be included in a variety of research papers. But, before building the experiment, we must first determine what it will do. For this tutorial, I have chosen to apply various classifiers on discretized data sets to determine which combination of methods performs the best using varying criteria (explained in Section \ref{evaling}). 

We must first choose an executable file that contains our shell code used in the experiment. For now, this can be named $my$-$first$-$experiment.sh$, located anywhere seen fit. Once the file is created and can be executed, the required scripting code can then be added to it.

First, let's construct the function that will be the main experiment. For this, I have named the experiment as $exp1$, and the structure should resemble the following:

{\small
\begin{verbatim}
exp1(){
	       #place experiment code here...	
}
\end{verbatim} 
}

This simple structure will contain all of the operations of our experiment. $exp1$ will, however, call other already-defined functions within OURMINE (such as the learners, splitting training and testing sets, etc.). 

For the next step, we need to declare local variables used in the experiment. These variables will make it easy to change certain parameters of the experiment later on, such as the number of runs and bins, the learners or data sets used, and others. So now we will declare the required variables for the experiment. These are $runs$, $bins$, $datas$, and $learners$. $runs$ will tell us how many times we wish to perform a cross-validation. For instance, a 10 X 10-way cross-validation means that we wish to perform a 10-way cross-validation $ten$ times. $bins$ tells us how the cross-validation is performed -- using 10 bins means testing on $\frac{1}{10}$-th and training on $\frac{9}{10}$-ths of the data, while using 5 bins means testing on $\frac{1}{5}$-th and training on $\frac{4}{5}$-ths. $datas$ simply lists the data sets that we wish to use in the experiment. $learners$ lists the learners used in the experiment.

The experiment code is now

{\small
\begin{verbatim}
exp1(){
	    local runs=10
	    local bins=10
	    local datas="audiology.arff mushroom.arff vote.arff soybean.arff"
	    local learners="zeror oner nb j48"
}
\end{verbatim} 
}

Now that the required variables are set, we will need to construct the "meat" of the experiment by first building the data loop, and discretizing each data set before continuing. 

{\small
\begin{verbatim}
exp1(){
	    local runs=10
	    local bins=10
	    local datas="audiology.arff mushroom.arff iris.arff vote.arff soybean.arff"
	    local learners="zeror oner nb j48"
	
	    for data in $datas; do
    done
}
\end{verbatim}
}


Here I am iterating through each data set in the, located in the same directory as the experiment file for simplicity. Now I add the ability to execute each learner

{\small
\begin{verbatim}
exp1(){
	    local runs=10
	    local bins=10
	    local datas="audiology.arff mushroom.arff vote.arff soybean.arff"
	    local learners="zeror oner nb j48"
	
	    for data in $datas; do
             for learner in $learners; do
        
       	    done
    done
}
\end{verbatim}
}
Next, we wish to add the functionality of performing a cross-validation. In this experiment, I am going to use the standard 10X10-way for consistency. First, we must add the run and bin loops required to perform the cross-validation a certain number of times. 
\newpage
{\small
\begin{verbatim}
exp1(){
	   local runs=10
	   local bins=10
	   local datas="audiology.arff mushroom.arff vote.arff soybean.arff"
	   local learners="zeror oner nb j48"
	
	   for data in $datas; do
           for learner in $learners; do
        	    for((run=1;run<=$runs;run++)); do
            	    for((bin=1;bin<=$bins;bin++)); do
					
	    		            done
        	    done
           done
    done
}
\end{verbatim}
}
Each learner now has to be trained using a $training$ set, and tested using a $testing$ set. These are built using the aforementioned OURMINE function {\bf makeTrainAndTest}. {\bf makeTrainAndTest} builds a training set and testing set from the original data set using parameters passed to it. Remember from Section \ref{tandt} that we can tell the function how to construct the training and testing sets by supplying it with the bin number. Also recall that train.arff and test.arff are placed into the current working directory.
\newpage
{\small
\begin{verbatim}
exp1(){
	   local runs=10
	   local bins=10
	   local datas="audiology.arff mushroom.arff vote.arff soybean.arff"
	   local learners="zeror oner nb j48"
	
	   for data in $datas; do
           for learner in $learners; do
        	    for((run=1;run<=$runs;run++)); do
            	    for((bin=1;bin<=$bins;bin++)); do
				                     makeTrainAndTest $data $bins $bin
                     goals=`cat train.arff | getClasses --brief`
                     $learner train.arff test.arff | gotwant > result.dat

	    		            done
        	    done
           done
    done
}
\end{verbatim}
}
There are two parts of this newer expansion of the experiment that have yet to be discussed. The first is the $goals$ variable. $goals$ is set to the result of the sub-shell giving all of the classes (using the built-in function {\bf getClasses} from the newly created training set. These represent all classes we wish to focus on for the proceeding testing phases.

Running a learner on a training and testing set has been shown before. However, the {\bf gotwant} function has not. {\bf gotwant}
outputs results in the form of classes that were predicted by the classifier for the test instance, as well as the actual classes for that test instance. These values are then used to build the confusion matrix using the {\bf abcd} function, like the following:

{\small
\begin{verbatim}
exp1(){
	   local runs=10
	   local bins=10
	   local datas="audiology.arff mushroom.arff vote.arff soybean.arff"
	   local learners="zeror oner nb j48"
	
	   for data in $datas; do
           for learner in $learners; do
        	    for((run=1;run<=$runs;run++)); do
            	    for((bin=1;bin<=$bins;bin++)); do
				                     makeTrainAndTest $data $bins $bin
                     goals=`cat train.arff | getClasses --brief`
                     $learner train.arff test.arff | gotwant > result.dat                    
                     for goal in $goals; do
                        cat result.dat |
                        abcd --prefix "$data,$learner,$bin,$run,$goal" \
                             --goal "$goal" \
                             --decimals 1
                    done
                    blabln "$data,$learner,$bin,$run,$goal"
	    		            done
        	    done
           done
    done
}
\end{verbatim}
}

Each goal is passed to the {\bf abcd} function, along with the output from the classification stage. The function then builds the confusion matrix from these results. The {\bf --prefix} passed to the function simply states that the contents of the string will be printed before the statistics from the matrix. By executing the {\bf show abcd} command, we can see that the values printed by the function are: {\em a, b, c, d, accuracy, pd, pf, precision, balance}. Thus, a single call to the function as written in the above experiment code could produce something such as 

%\begin{center}
{\small
	\begin{verbatim}
audiology.arff, nb, 5, 1, cochlear_unknown, 17, 1, 0, 4, 95.5, 80.0, 0.0, 100.0, 85.9
	\end{verbatim}
}
%\end{center}

these results will eventually be concatenated into one file for further and easy analysis. {\bf blabln} was also added to the experiment to provide the user with a quick update on the current status of the experiment.

The experiment is nearly complete. However, we still must place all of the experiment's output to a properly formatted holding file. This is accomplished by encapsulating the results in memory until being finally written to the output file. While this gives us a rapid way to store results, it is not recommended for experiments that require large amounts of time. In the event of a machine crash or power outage, incrementally saving results to files allows for the ability to restart the experiment at a desired location without the need for a completely fresh run. For the current example, however, this is adequate.
\newpage
{\small
\begin{verbatim}
exp1(){
	   local runs=10
	   local bins=10
	   local datas="audiology.arff mushroom.arff vote.arff soybean.arff"
	   local learners="zeror oner nb j48"
	   
	   (echo "#dataset,learner,bin,run,goal,a,b,c,d,acc,pd,pf,prec,bal"
	   for data in $datas; do
           for learner in $learners; do
        	    for((run=1;run<=$runs;run++)); do
            	    for((bin=1;bin<=$bins;bin++)); do
				                     makeTrainAndTest $data $bins $bin
                     goals=`cat train.arff | getClasses --brief`
                     $learner train.arff test.arff | gotwant > result.dat
                     
                     for goal in $goals; do
                        cat result.dat |
                        abcd --prefix "$data,$learner,$bin,$run,$goal" \
                             --goal "$goal" \
                             --decimals 1
                    done
                    blabln "$data,$learner,$bin,$run,$goal"
	    		            done
        	    done
           done
    done) | malign > exp1-output.csv
}
\end{verbatim}
}

Note the addition of a header that is given to the output file (exp1-output.csv) describing the meaning of the column values. This is crucial if others are to be looking at the results and attempting to analyze them. Also see the use of {\bf malign} to neatly organize the results.

The addition of one extra line is required to run the experiment.

{\small
\begin{verbatim}
exp1(){
	   local runs=10
	   local bins=10
	   local datas="audiology.arff mushroom.arff vote.arff soybean.arff"
	   local learners="zeror oner nb j48"	
	   
	   (echo "#dataset,learner,bin,run,goal,a,b,c,d,acc,pd,pf,prec,bal"
	   for data in $datas; do
           for learner in $learners; do
        	    for((run=1;run<=$runs;run++)); do
            	    for((bin=1;bin<=$bins;bin++)); do
				                     makeTrainAndTest $data $bins $bin
                     goals=`cat train.arff | getClasses --brief`
                     $learner train.arff test.arff | gotwant > result.dat
                     
                     for goal in $goals; do
                        cat result.dat |
                        abcd --prefix "$data,$learner,$bin,$run,$goal" \
                             --goal "$goal" \
                             --decimals 1
                    done
                    blabln "$data,$learner,$bin,$run,$goal"

	    		            done
        	    done
           done
    done) | malign > exp1-output.csv
}

exp1
\end{verbatim}
}
The experiment is now finished. To run the experiment from within the OURMINE environment, navigate to the directory containing $my$-$first$-$experiment.sh$, and enter

\begin{verbatim}
OURMINE> . first-experiment.sh
\end{verbatim}

This concludes the building of the experiment. Please note that many more complex tasks can be accomplished in much the same way in OURMINE by using both custom and built-in functions. In the Section \ref{evaling}, I analyze the results from this experiment using statistical analyses tools (also built into OURMINE).

\subsection{Evaluating Results using Ourmine}\label{evaling}

There are two main ways discussed in this thesis to analyze classification results. The first way, which I've already demonstrated, is through the use of quartile charts. As previously discussed, quartile charts are used to represent minimum, maximum, median and variance of certain evaluation measures. However, in order to determine if these values are {\em statistically} different, a statistical test is required. In this section, I will discuss (briefly) the statistical tests used in OURMINE to {\em rank} treatments, and then give examples as to how these tests are conducted using OURMINE on the results from the previous section.

The Mann-Whitney test (also referred to as the Mann-Whitney U test) is a rank sum, statistical test used to determine if values in two unpaired samples of observations are equally large. For an evaluation measure that is to be maximized, such as $pd$, this is essential because it gives us information regarding the size of values making up the entire distribution. 

Running the Mann-Whitney test is simple in OURMINE, but there are a few parameters that must be passed to the function in order for it to work properly. An example of how this can be conducted on the output of the experiment in Section \ref{building-exp}.

{\small
\begin{verbatim}
OURMINE> winLossTie --input exp1-output.csv --test mw --fields 14 --perform 11 --key 2
\end{verbatim}
}

The parameters are as follows:

\bi
\item{{\bf --input}: simply gives the file containing the results from the experiment in a comma-separated form}
\item{{\bf --test}: provides the script with the test to be used. These can either be $mw$ for the Mann-Whitney test, or $w$ for the Wilcoxon test (described below)}
\item{{\bf --fields}: the total number of fields in the output file}
\item{{\bf --perform}: provides the function with the column number in which to perform the statistical test, from left to right. In this case, field 11 points to the column giving {\em pd} values.}
\item{{\bf --key}: provides the function with the column number of the key to be used during the statistical test. In this experiment, the key is the learner to be tested. Since the output file prints the data set and then the learner, the key is then 2} 
\ei

Thus, the results obtained from the above is as follows:

{\small
\begin{verbatim}
  #key, ties, win, loss, win-loss
   j48,    1,   2,    0,        2
    nb,    3,   0,    0,        0
 zeror,    2,   0,    1,       -1
  oner,    2,   0,    1,       -1
\end{verbatim}
}


where the {\bf \#key} is the learner. Ties are labeled in the second column, and detail how many treatments (in this case, learners), that particular treatment tied with in the comparison of distributions. For example, {\em j48} tied with only one another treatment. However, the decision tree classifier won against another two, and lost against none. Therefore, the end result is in the last column represented by the difference in the number of wins and losses. Here, it can be shown that {\em j48} performed the best, followed by {\em nb}. {\em oner} and {\em zeror}, however, obtained the same value, and so can be grouped together and considered as having the same rank.

Note that while this ranking works in comparing equally {\em large} values, such as those that are to be maximized, there is no problem. However, when ranking measures that are to be minimized, such as {\em pf}, for instance, the performance column is to be negated. In other words, by doing this, normally large values (great for $pd$) are made to be very small, and very small values are made to be large. Since the Mann-Whitney test looks at the size of the values in these distributions and ranks based on largeness, previously large values (bad for $pf$) will not fair as well as previously small values, which are now large. 

Another way to accomplish this for minimized values is by simply negating the values in the {\em win-loss} column. By doing this, rankings are reversed. However, this may not be ideal, as ranking can be based on whatever the experimenter chooses, such as $wins$, $losses$, or $wins-losses$. 

The Wilcoxon test is used to rank {\em paired} samples of observations from an experiment's results. For example, if in Section \ref{building-exp}, the learners in the experiment would have shared testing sets at classification time, the Wilcoxon test would be performed on the results. 

To run the Wilcoxn test in OURMINE is as easy as it is to run the Mann-Whitney test. An example of this would be as follows -- almost identical to the previous test, but with an adjustment in the $test$ parameter.

{\small
\begin{verbatim}
OURMINE> winLossTie --input exp1-output.csv --test w --fields 14 --perform 11 --key 2
\end{verbatim}
}
Note the slight change from {\bf --test mw} to {\bf --test w}. 