%\documentclass[onecolumn,draftcls,journal,compsoc]{IEEEtran}
%\documentclass[technote]{IEEEtran}
\documentclass{sig-alternate}
\usepackage{graphicx}
\usepackage{wrapfig,url}
\newcounter{over}
\newcounter{max}
\newcommand{\fred}[2]{\setcounter{over}{#2}\addtocounter{over}{#1}}
\newcommand{\boxplot}[5]{%
\scalebox{0.5}{\textcolor{white}{\setcounter{max}{#4}\addtocounter{max}{#5}\fred{#4}{-#2}}%
\begin{picture}(100,10)%
\put(0,0){\line(0,1){8}}%
\put(100,0){\line(0,1){8}}%
%\put(#1,4){\circle*{1}}%
%\put(\themax,4){\circle*{1}}%
\put(#2,4){\line(1,0){\theover}}%
\put(#3,4){\circle*{8}}%
%\put(#4,4){\line(1,0){#5}}%
\put(50,0){\line(0,1){8}}%
\end{picture}}
} 
\usepackage{eso-pic}
\usepackage{color}
\usepackage{type1cm}
%\makeatletter
%  \AddToShipoutPicture{%
%    \setlength{\@tempdimb}{.5\paperwidth}%
%    \setlength{\@tempdimc}{.5\paperheight}%
%    \setlength{\unitlength}{1pt}%
%      \makebox(600,900){\rotatebox{45}{\textcolor[gray]{0.80}{\fontsize{5cm}{5cm}\selectfont{Draft v5}}}}}
%\makeatother
%
\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{itemize}
   \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{itemize}}
\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}

 \newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}
 \newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
 \newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}

%\usepackage{times}
\usepackage{alltt}
\usepackage{url}
% IEEE Computer Society needs nocompress option
% requires cite.sty v4.0 or later (November 2003)
\usepackage{cite}
% normal IEEE
 % \usepackage{cite}

% correct bad hyphenation here
\hyphenation{op-tical net-works semi-conduc-tor}


\pdfpagewidth=8.5in
\pdfpageheight=11in
\begin{document}

%\conferenceinfo{PROMISE'09,} {May 18--19, 2009, Vancouver, Canada} 
\conferenceinfo{~} {~} 

\CopyrightYear{2009}

\crdata{978-1-60558-036-4/08/05}


%\CopyrightYear{2008} % Allows default copyright year (200X) to be over-ridden - IF NEED BE.
%\crdata{0-12345-67-8/90/01}  % Allows default copyright data (0-89791-88-6/97/05) to be over-ridden - IF NEED BE.

\title{How to Build Repeatable Experiments}
%\titlenote{Submitted to the PROMISE 2008 workshop on repeatable SE experiments.
%%A draft version of this paper may be downloaded fro
%\url{http://menzies.us/pdf/08keys.pdf}
%}

\numberofauthors{2}
%
\author{
%% 1st. author
\alignauthor Gregory Gay, Tim Menzies, Bojan Cukic
\\
       \affaddr{CS\& EE, WVU, Morgantown, WVU}\\
       \email{gregoryg@csee.wvu.edu, tim@menzies.us, cukic@csee.wvu.edu}
\alignauthor Burak Turhan\\
       \affaddr{NRC Institute for Information Technology}\\
       \affaddr{Ottawa, Canada}\\
	\email{Burak.Turhan@nrc-cnrc.gc.ca}	
}
%
\date{12 January 2009}

\maketitle
\begin{abstract}
The mantra of the PROMISE series is ``repeatable, improvable, maybe refutable''
software engineering experiments. 
This community has successfully created a library
of reusable software engineering data sets.
The next challenge in the PROMISE community will be to not only share data, but to share
experiments.
Our experience with existing data mining environments is that
these tools are not suitable for  
publishing or sharing repeatable experiments. 

OURMINE is an environment for the development of data mining
experiments. OURMINE offers a succinct notation for describing experiments.
Adding new tools to OURMINE, in a variety of languages,
is a rapid and simple process.  This makes it a useful research
tool. Complicated graphical
interfaces have been eschewed for  simple command-line prompts.
This simplifies the learning curve for data mining novices.  The
simplicity also encourages large scale modification and experimentation
with the code.

In this paper, we show the
OURMINE code required to reproduce a recent experiment
checking
how defect predictors learned from one site apply to another.  This
is an important result for the PROMISE community since it shows
that our shared repository is not just a useful academic resource.
Rather, it is a valuable resource industry: companies that lack the
local data required to build those predictors can use PROMISE data
to build defect predictors.
\end{abstract}
\category{i.5}{learning}{machine learning}
\category{d.2.8}{software engineering}{product metrics}
\keywords{algorithms,experimentation, measurement}

\newpage

\section{Introduction}

With the recent rise of powerful open-source development platforms also came
the development of data mining environments that offered seamless connection between
a wide range of tools. The most famous of these tools is the WEKA\footnote{\url{http://www.cs.waikato.ac.nz/ml/weka/}} ~\cite{witten05} from the
Waikato University's Computer Science Machine Learning Group (see \fig{weka}). WEKA contains
hundreds of data miners, has been  widely used
for teaching and experimentation, and won the
the 2005 SIGKDD Data Mining and Knowledge Discovery Service Award.
Other noteworthy tools are ``R''\footnote{\url{http://www.r-project.org/}}, ORANGE\footnote{\url{http://magix.fri.uni-lj.si/orange/}} (see \fig{orange}) and MATLAB \footnote{\url{http://www.mathworks.com}}
 (since our interest is in the ready
availability of tools, the rest of this paper will ignore proprietary tools such as MATLAB).

Our complaint with these tools is the same issue raised by
Ritthoff et al.~\cite{ritthoff01}. Real-world data mining 
experiments  (or applications)
are more complex that 
running one algorithm. Rather, such  experiments or applications require {\em intricate combinations}
of a large 
number of tools that include data miners, data pre-processors and report regenerators.
Ritthoff et al. argue  (and we agree) that
the standard interface of (say) WEKA  does not support the rapid generation of
these intricate combinations.

The need to ``wire up'' data miners within a multitude of other tools has been addressed
in many ways. In WEKA's  visual {\em knowledge flow} programming environment, for example,
nodes represent different pre-processors/ data miners/ etc, and arcs represent data flows between them.
A similar visual environment is offered by many other tools including ORANGE (see \fig{orange}).
The YALE tool (now RAPID-I) built by Ritthoff et al. formalize these visual environments by generating
them from  XML
schema describing
{\em operator trees} like \fig{yale}.

\begin{figure}[!t]
\begin{center}
\includegraphics[width=2.25in]{weka.jpg}
\end{center}
\caption{WEKA.}\label{fig:weka}
\end{figure}
\begin{figure}[!t]
\begin{center}
\includegraphics[width=2.25in]{orange.jpg}
\end{center}
\caption{Orange.}\label{fig:orange}
\end{figure}
\begin{figure}
\begin{center}
\includegraphics[width=2in]{yale.png}
\end{center}
\caption{Data mining operator trees. From ~\cite{mie06}.}\label{fig:yale}
\end{figure}

In our experience in CS, students find these
visual environments either
discouraging or distracting:
\bi
\item Some are discouraged by the tool complexity. These students shy away
from extensive modification and experimentation.
\item Others are  so enamored with the impressive software engineering
inside these tools that they waste an entire semester building environments to support data mining,
but never get around to the data mining itself.
\ei
A similar experience is reported by our colleagues in the WVU statistics department.
In order to support the learning process, they 
bury these data mining tools inside high-level scripting environments. Their scripting environments
shields the user from the complexities of ``R''
by defining some high-level LISP code that, internally, calls ``R.''

Our experience with existing data mining toolkits is that
these tools are not suitable for  
publishing or sharing repeatable experiments.  This paper discusses why we believe this to be so and offers OURMINE as an alternate
format for sharing executable experiments.

OURMINE is a data mining environment used at West Virginia University
to teach data mining as well as to support the  generation of  our research
papers. OURMINE does not replace WEKA, ``R'', etc. Rather, it takes a UNIX shell scripting
approach that allows for the  ``duct taping'' together of various tools. For example,
OURMINE currently calls WEKA learners as sub-routines, but it is not limited to that tool.
This is no requirement to code in a single language
(e.g. the JAVA used in WEKA) so data miners written in ``C'' can be run along side
those written in JAVA or any other language.
Any program with a command-line API can be combined with other programs within OURMINE. In this way,
we can lever useful components from WEKA, R, etc,  while quickly adding in scripts
to handle any missing functionality. 

Based on three years experience with OURMINE,
we assert that the tool has certain pedagogical advantages.
The simplicity of the tool encourages experimentation
and modification, even by data mining novices.
We also argue that OURMINE is a productive
data mining research environment. For example $\frac{9}{10}$ of the first
and second
authors' recent papers used OURMINE.

More importantly, OURMINE offers a succinct
executable representation of an experiment that can be included in a research paper.
To demonstrate that, we present:
\bi
\item 
The OURMINE script reproducing
a recent SE experiment~\cite{turhan08}. 
\item
The results obtained from that script.
\ei
The experiment selected is of particular
importance to the PROMISE community since it shows that PROMISE data sets collected
from one source can be applied elsewhere to create good defect predictors. This in turn
means that the PROMISE repository is not just a useful academic resource, but is also
a valuable source of data for industry. If a company wants to build defect predictors, but
lacks the local data required to build those predictors, then by using the right 
{\em relevancy filters} (described below) it is possible to use data imported from the PROMISE
repository to generate defect predictors nearly as good as those that might be built
from local data.


\section{OURMINE}
While a powerful language, we find that LISP can be 
arcane to many audiences. Hence, OURMINE's script are a combination of BASH ~\cite{ramey94} and GAWK ~\cite{awkbook}.
Our choice of BASH/GAWK over, say, LISP is partially a matter of
taste but we defend that selection as follows.  Once a student
learns RAPID-I's XML configuration tricks, then those learned
skills are highly specific to that particular tool.  On the other
hand, once a student learns BASH/GAWK methods for data pre-processing
and reporting, they can apply those scripting tricks to any number
of future applications.

\begin{figure}
~\hrule~
{\scriptsize\begin{verbatim}
#naive bayes classifier in gawk
#usage:  gawk -F, -f nbc.awk Pass=1 train.csv Pass=2 test.csv

Pass==1 {train()}
Pass==2 {print $NF "|" classify()}

function train(    i,h) { 
   Total++;
   h=$NF;     # the hypotheis is in the last column
   H[h]++;    # remember how often we have seen "h"
   for(i=1;i<=NF;i++) {
     if ($i=="?") 
           continue;         # skip unknown values
     Freq[h,i,$i]++
     if (++Seen[i,$i]==1) 
           Attributes[i]++}  # remember unique values
}

function classify(         i,temp,what,like,h) {  
   like = -100000;         # smaller than any log
   for(h in H) {           # for every hypothesis, do...  
     temp=log(H[h]/Total); # logs stop numeric errors
     for(i=1;i<NF;i++) {  
       if ( $i=="?" ) 
             continue;     # skip unknwon values
       temp += log((Freq[h,i,$i]+1)/(H[h]+Attributes[NF])) }
     if ( temp >= like ) { # we've found a better hypothesis
        like = temp
        what=h}
   }
   return what;
}\end{verbatim}}
~\hrule~
\caption{A Naive Bayes classier for data sets in csv format where the last column is the class.}\label{fig:nbc}
\end{figure}
\begin{figure}
\begin{tabular}{p{.95\linewidth}}\hline
\small
\be
\item Modify \fig{nbc} so that there is no train/test data. Instead, make it an incremental learner.
Hint: 1) call the functions $train$, then $classify$ on every line of input. 2) The order is important: always
$train$ before $classifying$ so the the results are always on unseen data.
\item Convert \fig{nbc} into HYPERPIPES~\cite{Eisenstein04}. Hint: 1) add globals $Max[h,i]$ and $Min[h,i]$ to 
keep the max/min values seen in every column ``$i$'' and every hypothesis class ``$h$''. 2) 
Test instance belongs to the class that most overlaps the attributes in the test instance.
So, for all attributes in the test set, sum the returned values from $contains1$:
{\scriptsize\begin{verbatim}
function contains1(h,i,val,numerip) {
  return numericp ?  
		        Max[h,i] >= val && Min[h,i] <= val : 
	        (h,i,value) in Seen }\end{verbatim}}
\item Use \fig{nbc} for anomaly detector. Hint:  1)~make all training examples classify as the same class;
2) an anomalous test instance has a likelihood $\frac{1}{\alpha}$ of the mean likelihood
seen during training ($alpha$ needs tuning but $alpha=50$ is often useful).
\item Using your solution to \#1, create an  incremental version of HYPERPIPES and an anomaly detector.
\ee
\\\hline
\end{tabular}
\caption{Four Introductory OURMINE programming exercises.}\label{fig:ourmine101}
\end{figure}
Another reason to prefer scripting over the complexity of RAPID-I, WEKA, ``R'', etc,
is that it reveals
the inherent
simplicity of many of our data mining methods. For example, \fig{nbc} shows a GAWK implementation of
a Naive Bayes classifier for discrete data where the last column stores the class symbol.
This tiny script  is no mere toy- it successfully executes on very large
datasets such as those seen in the 2001 KDD cup. WEKA, on the other hand, cannot process these large
data sets as it always loads its data into RAM. \fig{nbc}, on the other hand, only requires
enough memory to store one instance as well as the frequency counts in the hash table ``$F$''.

More importantly, in terms of teaching, \fig{nbc} is easily customizable. 
For example, \fig{ourmine101} shows four warm-up exercises for novice data miners that (a)~introduce them
to basic data mining concepts and (b)~show them how easy it is to script their own data miner:
Each of these tasks requires changes to fewer than 10 lines in \fig{nbc}. 
The ease of these customizations fosters a spirit
of ``this is easy'' for novice 
data miners. This, in turn, empowers them to design their own extensive and elaborate experiments.
\subsection{Built-in Data/Functions}
The appendix of this paper describes the download and install instructions for
OURMINE. The standard install  comes with the following public domain data sets (found in the \$OURMINE/lib/arffs directory):
\bi
\item PROMISE:  cm1, kc1, kc2, kc3, mc1, mc2, mw1, pc1, pc2, pc3, pc4, pc5, ar3, ar4, ar5
\item UCI (discrete): anneal, colic, hepatitis, kr-vs-kp, mushroom, sick, waveform-5000, audiology, credit-a, glass, hypothyroid, labor, 
pcolic, sonar, vehicle, weather, autos, credit-g, heart-c, ionosphere, letter, primary-tumor, soybean, vote,\newline weather.nominal,
breast-cancer, diabetes, heart-h, iris, lymph, segment, splice, vowel
\item UCI (numeric): auto93, baskball, cholesterol, detroit, fruitfly, longley, pbc, quake, sleep, autoHorse, bodyfat, cleveland, 
echoMonths, gascons, lowbwt, pharynx, schlvote, strike, autoMpg, bolts, cloud, elusage, housing, mbagrade, pollution, sensory, veteran, 
autoPrice, breastTumor, cpu, fishcatch, hungarian, meta, pwLinear, servo, vineyard
\ei

\begin{figure}[!t]
\small
~\hrule~
\begin{description}
\item[buildIncrementalSet]  builds a set containing a set number of instances;  
\item[classes]find all values for a class attribute;
\item[combineFilesRandom] combines two data sets, randomizing the lines;
\item[combineFiles] combines two data sets with common attributes
\item[gains]runs InfoGain on  the data 
\item[intersectAttributes] finds the intersection of attributes over several data sets;
\item[logNumbers] logs all numeric values in a data set ;
\item[randomizeFiles] randomizes lines in a data set;
\item[rankViaInfoGain] ranks attributes by InfoGain values;
\item[removeAttributes] performs column pruning;
\item[sample] creates an over or under-sampled dataset;
\item[shared] find shared attributes;
\item[someArff] splits a data set into $train.arff$ and $test$.arff where the test fail contains the B-th division of $\frac{1}{B}$-the data and train file contains the rest.
\item[some] generates an ARFF file containing certain attributes;
\end{description}
~\hrule~
\caption{OURMINE data manipulation functions.}\label{fig:functions}
\end{figure}
\begin{figure}
\small
~\hrule~
\begin{description}
\item[abcd:] finds a,b,c,d values and reports pd, pf, accuracy, precision, and balance.
If $\{A,B,C,D\}$ are the
true negatives,
false negatives, 
false positives, and 
true positives
(respectively) found by a defect predictor, then:
\[
\begin{array}{rl}
pd=recall=&D/(B+D)\\ 
pf=&C/(A+C)\\ 
balance=&1 - \frac{\sqrt{\left(0 - pf\right)^2 + \left(1 - pd\right)^2}}{\sqrt{2}}
\end{array}\]
\item[columnStats:] find column min, max, mean, and std.deviations.
\item[medians:] finds medians;
\item[quartiles:] generates box plots showing quartiles;   
\item[uniques:] finds the  unique entries in a column;
\item[winLossTie:] runs the Mann-Whitney test
\end{description} 
~\hrule~
\caption{OURMINE statistical functions.}\label{fig:stats}
\end{figure}

OURMINE also comes with  a library of common functions used in machine learning experiments.
All learners are the WEKA implementations. The functions that interface OURMINE with the WEKA set
certain command-line parameters. To see those flag settings, type {\em show <learner name>} at 
the command prompt.  That library includes:
\bi
\item The data manipulation functions of \fig{functions};
\item The statistical functions of \fig{stats};
\item Some reporting functions such as code to auto-generate the latex
files required to report quartile charts. For examples of those charts, see \fig{an2PD} and
\fig{an2PF} (later in this article).
\item Learners: oner, jRip, jRip10, part, aode, aode10, nb (naive bayes), nb10, nbk, lwl, j48, j4810, j4810c, lsr,
m5p, 1Bkx, 1Bk, apriori

\ei
\subsection{Extending the Functionality}
OURMINE's library contains the 
functions used by a team of
12 graduate students and one faculty student researching data mining over a three
year period.
This library is hardly complete and
one of the goals of this paper is to create a community of OURMINE programmers so that
the library of functions can be extended.

Adding new functionality to OURMINE is
not difficult. OURMINE's library of scripts are contained in 
a file called {\em minerc} which is just
a collection of functions written in the BASH scripting language.  Hence, to add functionality:
\bi
\item Create a separate BASH file (i.e. {\em functions.sh}).
\item Write all new functions in this file.
\item At the end of {\em minerc}, add the line ``{\em . functions.sh}''. 
\ei

Very little knowledge of BASH is required to add functions to OURMINE. If the new function is 
actually in a JAVA file, the BASH function could just instantiate the Java file. 
In \fig {example1}, for example,
the BASH function $nb$ uses a series of commands to instantiate a JAVA classifier. The {\em blab} is a built-in function that prints text to the screen. The second 
line is the command to launch an external Java application (-Xmx1024M is a flag that gives the Java application a certain 
amount of memory). The process to call external programs is similar for any language. 

\begin{figure}
~\hrule~
{\scriptsize
\begin{verbatim}
nb() {	
        blab "n"
        java -Xmx1024M -cp $Tmp/weka.jar \
                 weka.classifiers.bayes.NaiveBayes \
                -p 0 -t  $1 -T $2  
}
\end{verbatim}}
~\hrule~
\caption{OURMINE  calling the WEKA}
\label{fig:example1}
\end{figure}
The majority of the functions in OURMINE are BASH  scripts 
that glue together GAWK code.
GAWK is simple to learn
and encourages short, easily modifiable, programs.  
R. Loui, an associate professor in Computer Science at Washington University in St. Louis, 
strongly supports the use of GAWK in his artificial intelligence classes. He writes:
\begin{quote}{\em
There is no issue of user-interface. 
This forces the programmer to return to the question of what the program does, not how it looks. 
There is no time spent programming a binsort when the data can be shipped to /bin/sort in no time.} ~\cite{gawkai}
\end{quote}
\begin{figure}
~\hrule~
{\scriptsize
\begin{verbatim}
     1	someData() {
     2	    cat $1 | gawk '
     3	    BEGIN           { IGNORECASE =1                
     4	                      srand('$RANDOM')            
     5	                      N='$2' }                   
     6	    gsub(/%.*/,"")           
     7	    /^[ \t]*$/      { next }                   
     8	    In              { Data[rand()]=$0 }       
     9	    /@data/         { In=1 }                 
    10	    /@/             { next }                
    11	    END             { for(D in Data) {    
    12	                         print Data[D]   
    13	                         if ((--N) <= 0) exit }} '
    14	}\end{verbatim}}
~\hrule~
\caption{SomeData: An OURMINE function }
\label{fig:example2}
\end{figure}
For example, the GAWK script of \fig {example2} 
extracts $N$ randomly selected data lines from an arff file.
It uses
two parameters as input: {\em \$1=input file};  
{\em \$2=N} (how many lines of data we require).
\bi
\item Lines 4 and 5 initializes random numbers  and $N$.
\item
Lines 6 and 7 are the standard idiom for ``skip lines that
are all blanks or comments''. If, after deleting all characters after comment,
the line contains only white space then we skip that line.
\item Line 8 stores a data line at a random location in the
$Data$ array, but only if Line 9 ran before and recognized the start of the data
section in this file (marked with  ``@data''). 
\item Line 10 skips all the meta lines
in the file.
\item Lines 10 through 13 print out $N$ random data items.
\ei
\subsection{Writing Experiments}
The OURMINE user writes experiments in a BASH notation
into a file loaded by
{\em minerc}. 
Running an experiment is as simple as typing the name of its function at the OURMINE command prompt (i.e. if your function is 
called {\em exp1}, it will run when you type {\em exp1} at the command prompt). 


\begin{figure}[!t]
~\hrule~
{\scriptsize
\begin{verbatim}
     1	demo15() {
     2	 cd $Tmp
     3	 (echo "#data,bin, a,b,c,d,acc,pd,pf,prec,bal" 
     4	  seed=$RANDOM;   
     5	  for((bin=1;bin<=10;bin++)); do
     6	     blab "$bin"
     7	     auto93discreteClass  | 
     8	            someArff --seed $seed --bins $Bins --bin $bin 
     9	     nb train.arff test.arff | 
    10	     gotwant                  | 
    11	     abcd --goal "_20"  --prefix "auto93,$bin" --decimals 1  
    12	   done | 
    13	   sort -t, -n -k 11,11
    14	   ) | 
    15	   malign > demo15.csv 
    16	   blabln " "
    17	   echo ""; cat demo15.csv
    18	   cp demo15.csv $Safe/demo15.csv
    19	   cd $Here
    20	}\end{verbatim}}
~\hrule~
\caption{An OURMINE experiment.}
\label{fig:bash}
\end{figure}
\fig {bash} shows a basic experiment from OURMINE.  When reading
BASH scripts like \fig{bash}, one thing to note is that the pipe
character ``|'' links the output of one process into the input of
another. Also, a ``|'' at the end of line means that the next line
inputs the piped output of the last line.

Line 5 of \fig{bash} shows that this a  10-way cross-validation
experiment, executed over one
file (see line 7: auto93 from the UCI
collection).  At line 8, the  $someArff$ function splits the data
into "train.arff" (containing 90\% of the data) and "test.arff"
(containing 10\% of the data).

At line 9, a Naive Bayes learner of \fig{example1} is called. 
A more general construct, seen in the experiment described below is to loop through, a set 
of learners using 
\[
\mbox{{\em for learner in \$learners; do}}
\]in which case, line 9 would become
\[
\mbox{{\em \$learner train.arff test.arff}}
\]
The code in lines 10 and 11 collect the performance statistics and print out
$a, b, c, d, acc, pred, pf, prec, bal$.  Some of these statistics
are class-dependent. Line 11 shows the idiom {\em abcd --goal "\_20"} which instructs
OURMINE to only report statistics for the class "\_20". If statistics for all
classes are required, then the OURMINE idiom is to cache the result of the learner,
then loop through all classes calling $abcd$ on each class in turn.  When using the
OURMINE tools, that idiom can be see in lines 20 through 29 of \fig{an2}.

The sort function of line 13 of \fig{example1} sorts these results on accuracy (column 11)
and the results are saved to a safety box $\$Safe$.
This final saving step is required because, by default, OURMINE cleans up after
itself as it exits. Unless result files are saved to some safe place, they will
be wiped.

This is a simple experiment, and it should be fairly easy to see the flow between statements. 
If the settings used for the parameters of a function are confusing, 
students learning OURMINE can run that function separately at the BASH prompt 
or look at its source code using the {\em show <function name>} command within OURMINE. 

One nuance, not readily apparent in \fig{bash},
is that these scripts are naturally parallizable. The idiom
\[
\mbox{{\em ssh me@someMachine.edu runThis}}
\]
logs into a machine and runs the script $runThis$.  The script
executes in some default, rather restricted, environment so $runThis$
must now how to create the paths and temp directories needed for
execution.  OURMINE is designed to automatically create its execution
environment so it is easy to write $runThis$ scripts based on
OURMINE. In a standard university environment, where students have
the same password on multiple machines, then very large experiments
can be farmed out over those CPUs by running: \[\begin{array}{c}
\mbox{{\em ssh me@someMachine1.edu runThis1}}\\ \mbox{{\em ssh
me@someMachine2.edu runThis2}}\\ \mbox{{\em ssh me@someMachine3.edu
runThis3}}\\ \mbox{{\em etc}} \end{array} \] This lets researchers
complete lengthy data mining experiments in minutes to hours, rather
than days to weeks.

%\section{OURMINE \& Teaching}
%
%We teach data mining with OURMINE in three stages:
%\be
%\item Introduction: Learning the basics of GAWK and BASH scripting, as well
%as learning the use of OURMINE. Due: week 4.
%\item Replication: Students replicate experiments using OURMINE. Due: week 7.
%\item Original Research: Students design and run their own experiments. Any work is 
%allowed, as long as it relates to the data mining concepts mentioned in class. Due: week 14.
%\ee
%In stage one, we offer programming assignments like:
%\bi
%\item
%Modify train/test code generator used in cross validation experiments so that the train file 
%comprises the first I bins and the test file contains just the I+1 bin (this is useful for incremental learning experiments). 
%\item
%Generate win/loss/tie tables using OURMINE's Mann-Whitney test to compare the performance of learning from 50,100,150,
%and 200 instances.
%\item 
%Modify OURMINE's Mann-Whitney script to become the Wilcoxon test.
%\ei
%In stage two, 
%students are asked to reproduce and understand
%the research of others. For example, we might ask them to
%reproduce Robert Holte's OneR vs j48 experiments ~\cite{holte93}; 
%Dougherty, Kohavi, and Sahami's look at various discretizers ~\cite{dou95}; or 
%the j48 vs RIPPER experiment conducted by Cohen ~\cite{cohen95}.
%Students work in groups and are asked  to succinctly summarize
%the research paper, showing their understanding of the techniques and conclusions. They were to then use OURMINE to reproduce, improve on, or refute 
%the original results of each paper. 
%
%This third and final stage tests the student's understanding of data mining concepts. Each group is asked  to design and conduct 
%their own experiment. They choose a topic and explore it in the allotted time. Some sample ideas are given, but students
%are free to do whatever they wanted (with instructor approval). In many cases, research students
%elect to perform the analysis of the data generated by their research work.
%Note that students are told that their experiments can fail to produce some improvement over the
%current state of the art in data mining. Such failures 
%were permitted as long as the students could prove that they understood why it failed. Each group is required to prepare a twenty-minute
%presentation and a conference-style research report based on their work. Grading of the project is based on the maturity and completeness 
%of the research work, measured against the Data Maturity Model \footnote{As defined by \url{http://www.csee.wvu.edu/~timm/cs591o/old/DMM.html}}. 
%Students have used OURMINE to explore areas such as dimensionality reduction, 
%modifications of existing learners, and nearest-neighbor clustering.  
%

\section{OURMINE \& Research}

To illustrate the use of OURMINE, the rest of this paper uses the
tool to reproduce a recent {\em Empirical Software Engineering}
paper written by the second and fourth authors~\cite{turhan08}.
Note that the following code was written by the first author with
minimal assistance from the others: for the most part, the first
author worked straight from the text of~\cite{turhan08}.

In their paper, Turhan et al.~\cite{turhan08} focus on binary defect prediction (defective or non-defective) and perform
three experiments to reach a conclusion in favor of either:
\bi
\item Cross-company (CC) data imported from other sites;
\item or within-company (WC)  data collected locally.
\ei
That paper made several conclusions:
\bi
\item {\bf Turhan\#1:} Using local WC data produces significantly better defect predictors than
using imported CC data.
\item {\bf Turhan\#2:} However, when {\em relevancy filtering} (see below)
is applied to the CC data, then the imported
CC data leads to defect predictors nearly as effective as using local WC data.
\item {\bf Turhan\#3:} Hence, while local data is the preferred option, it is feasible to use imported data
provided that it is selected by a relevancy filter.
\ei
This experiment is of great importance to the PROMISE community.
It shows that PROMISE data sets collected from 
one source can be applied elsewhere to learn good defect predictors; i.e.  
the PROMISE repository is not just a 
useful academic resource, but is also a valuable source of data for 
industry. Companies wishing to build defect predictors
can do so, even if they lack 
the local data required to build those predictors.

Such an important result needs confirmation.  Tools like
OURMINE are useful for such replication studies.
%     
%Ultimately, they (and we) are trying to answer the following questions:
%\bi
%\item Is cross-company data ever useful for organizations?:
%     Under which conditions, if any, is cross-company data preferable to within-company data for building
%defect predictor?

%\item Can companies filter CC data for local tuning?: 
%CC data includes information that may not pertain to local projects. The second experiment
%utilizes a simple form of nearest-neighbor filtering to build a subset of the CC data that
%is similar to the WC testing data.
%\item What is the smallest amount of local data needed for constructing a model?:
%  The third analysis focuses entirely on WC data in an attempt to determine the smallest amount of local
%  data needed for constructing a model. Incrementally larger training sets are used, then compared to 
%find the smallest set that does not lose to other increments. 
%\item Can these results be generalized?:
%The first three experiments only use data from NASA projects. The last analysis attempts to replicate
%the same conclusions using data from Turkish firm SOFTLAB as the WC data and the NASA sets as CC data. 
%Through this, the authors establish external validity.
%\ei
%

%\begin{figure}[!t]
%\begin{center}
%\small
%\begin{tabular}{p{.95\linewidth}}\hline
%%All these values range zero to one.
%%Better and {\em larger} balances fall  {\em closer} to the desired  zone 
%%of no false alarms and 100\% detection.\\\hline
%\end{tabular}
%\end{center}
%\caption{Performance measures.}\label{fig:measures}
%\end{figure}
%

\begin{figure}[!b]
%\begin{center}
%\begin{minipage}{.7\linewidth}
~\hrule~
{\scriptsize
\begin{verbatim}
 1	an2() {
 2	   local me="promise_an2"
 3	   local bins=10
 4	   local repeats=10
 5	   local learners="nb lwl"
 6	   local datas="PC1 CM1 KC1 KC2 KC3 MW1 MC2"
 7	   cd $Tmp
 8	   (echo "#data,repeat,bin,treatment,learner,goal,a,b,c,d \
 9	      ,acc,pd,pf,prec,bal"
10	   for((r=1;r<=$repeats;r++)); do
11	      for data in $datas; do
12	         arff=$OURMINE/lib/arffs/mdp/shared_$data.arff
13	         combined=$OURMINE/lib/arffs/mdp/combined_$data.arff
14	         blab "data=$data repeat=$r "
15	         seed=$RANDOM;
16	
17	         for((bin=1;bin<=$bins;bin++)); do
18	            blab "$bin"
19	            cat $arff | 
20	            someArff --seed $seed --bins $bins --bin $bin 
21	            goals=`cat $arff | classes --brief`
22	            for learner in $learners; do
23	               $learner train.arff test.arff | 
24	               gotwant > results.dat 
25	               for goal in $goals; do
26	                  cat results.dat |
27	                  abcd --goal "$goal" \
28	                    -prefix "$data,$r,$bin,WC,$learner,$goal" \
29	                    --decimals 1  
30	               done
31	            done                            
32	            goals=`cat $arff | classes --brief`
33	            blab "CC"
34	            local N=$[ ( $RANDOM % 10 ) +1 ]
35	            blab "($N)"
36	            cat $arff | 
37	            someArff --seed $seed --bins $bins --bin $N
38	            for learner in $learners; do
39	               makeTrainCombined $combined > trainCom.arff
40	               $learner trainCom.arff test.arff | 
41	               gotwant > results.dat 
42	               for goal in $goals; do
43	                  cat results.dat |
44	                  abcd --goal "$goal" \
45	                    --prefix "$data,$r,CC,CC,$learner,$goal" \
46	                    --decimals 1  
47	                done
48	             done  
49	
50	          done 
51	          blabln 
52	       done
53	    done       
54	   ) | sort -t, -n -k 12,12 | malign > $me.csv #each bin
55	   blabln " "
56	   echo ""; cat $me.csv
57	   cp $me.csv $Safe/$me.csv
58	   cd $Here
59	}
\end{verbatim}}
~\hrule~
%\end{minipage}
%\end{center}
\caption{WC or CC defect prediction. From~\cite{turhan08}.}
\label{fig:an2}
\end{figure}


\subsection{Experiment}

Many research papers in the field of data mining present results but bury the 
exact details of the implementation. After years of experience, it is clear that the majority
of time is spent in the tedium of pre and post-processing. The main contribution of Ourmine is to
provide functions to handle these tasks. 
The script shown in \fig{an2} is a complete structured experiment 
that is run in the Ourmine environment. What is not shown are the highly customized tools
that are frequently built and thrown away to deal with tasks like splitting a data set
into training and test subsets. An Ourmine function, {\em someArff} does this for the researcher. 
Many of the lines in this figure are actually references to internal Ourmine functions.

As a preliminary to this experiment, we took seven PROMISE defect
data sets (PC1,CM1,KC1,KC2,KC3,MW1,MC2) and built seven combined
data sets, each containing $\frac{6}{7}$-th of the data. For example,
the file: \[\mbox{{\em \$OURMINE/lib/arffs/mdp/combined\_PC1.arff}}\]
contains all seven data sets {\em except} PC1. This is the training
set used for the CC (cross-company) data.

Next, we conducted a 10*10-way cross validation experiment; i.e.
10 times we asked $someArff$ to randomly sort each data set, then
generate 10 train and test sets containing 90\% and 10\% of each
data set. The 90\% sets are the training set of the WC (within-company)
data.  Lines 17 to 31 of \fig{an2} shows that standard cross-validation
study.

The CC study is shown on lines 32 to 46 of \fig{an2}. This is the
same as the WC study but this time the training set is one the of the
$combined\_X.arff$ files described above.

The learners in this study were two Naive Bayes classifiers. $Nb$
(from \fig{example1}) and Frank et al.'s locally weighted Bayes
classifier called $lwl$~\cite{hallLWL}.  Naive Bayes classifiers
were used since: \bi \item Previously~\cite{me07b}, we have offered
evidence that they do better than other commonly used learners;
\item Lessmann et al. report that many other learners do no better
than Naive Bayes on the PROMISE defect data sets~\cite{lessmann08}.
\ei $Lwl$ applies {\em relevancy filtering} to the training set.
The idea behind relevancy filtering is to collect similar instances
together in order to construct a learning set that is homogeneous
with the testing set. Specifically, the aim of this filter is to
try to introduce a bias in the model by using training data that
shares similar characteristics with the testing data. Turhan et al.
implemented this with a combination of k-Nearest Neighbor (k-NN)
method and Naive Bayes.  For each test instance in the test set,
the k=10 nearest neighbors in the training set are collected.
Duplicates are removed, and the remaining examples are used to train
a Naive Bayes Classifier.


For reasons of repeatability, this study did not use Turhan et al.'s relevancy filter.
Rather, we used the $lwl$ from the WEKA. 
For each test instance, $lwl$
constructs a new naive Bayes model using a weighted subset of 
training instances in the locale of the test instance. 
In this approach, the size of the training subset is based on the distance of the kth nearest-neighbor to the instance
being classified. The training instances in this neighborhood are weighted, with more weight assigned to the closest instances. 
An instance identical to the test instance is given a weight of one, with the weight decreasing linearly to zero as the distance increases. 
Higher values for the number of neighbors ($k$) will result in models that are less sensitive to data anomalies, 
while smaller models will conform more closely to the data. The authors caution against using too small of a value for 
$k$, as it will 
be sensitive to the noise in the data. 
At the suggestion of  Frank et al. we ran \fig{an2} using $k\in \{10,25,50,100\}$.

\input{an2.tex}
\begin{figure}[!b]
\centering
\small
\begin{tabular}{r|l}
& Time spent \\
Step & (minutes)\\\hline
Prep work & ~38 \\
Data manipulation & 152 \\
Coding & 163\\
Experiment alteration (bugs, rework) & 210\\\hline
Total & $563 \approx 10$ hours
\end{tabular}
\caption{Time spent on each step} 
\label{fig:time2}
\end{figure}
\section{Discussion}
As \fig{time2} illustrates, the replication for this experiment was
completed within a matter of hours.  Original research would
undoubtedly take more time, but the tools in OURMINE clearly cut
the necessary time to set up an experiment, or to reproduce an experiment, by a large amount.

Note that the slowest task
in reproducing the experiment was the 
200 minutes for {\em experiment alteration}; i.e. finding and fixing
 conceptual bugs in version $i$ of the functions, then improve them
with $i+1$. In the past, we considered this work wasted time since it seemed that we were
just making dumb mistakes and correcting them. Now, we have a different view. It is
this process of misunderstanding, then correcting the details
of an experiment, that is the
real point of the experimentation:
\bi
\item If repeating an experiment takes too long,
we may abandon the
exercise and learn nothing from the process.
\item If repeating an experiment is too easy (e.g. just editing a RAPID-I operator tree)
then repeating the experiment does little to change our views
on software engineering and data mining.
\item But if repeating the experiment takes the {\em right} amount of time (not too long,
not too short),   and it challenges just
enough of our understanding of some topic, then we find that we ``engage'' with the problem;
i,e, we 
think hard about:
\bi
\item
The base assumptions of the experiment;
\item
 The interesting properties
of the data;
\item and the alternative methods that could achieve the same or better results.
\ei
\ei
That is, we advocate OURMINE and the ugly syntax of \fig{an2} over
beautiful visual programming environments like WEKA's knowledge
flow or the RAPID-I operator tree {\em because} the visual environments
are too easy to use (so we are not forced into learning new ideas).

On the other hand, we still want a productive
development environment  for developing our experiments.
It should be possible
to update a data mining environment and get new insights from the
tools, fast enough to keep ups motivated and excited to work on the
task.  In our experience, the high-level scripting of BASH and GAWK
lets us find that middle ground between too awkward and too easy.

\section{Conclusions}

The mantra of the PROMISE series is ``repeatable, improvable, maybe refutable''
software engineering experiments. 
This community has successfully created a library
of reusable software engineering data sets
that is growing in size and which have been used in multiple PROMISE papers:
\bi
\item 2006: 23 data sets
\item 2007: 40 data sets
\item 2008: 67 data sets
\item 2009: 85 data sets (at the time of this writing)
\ei

The next challenge in the PROMISE community will be to not only share data, but to share
experiments; 
i.e.  the PROMISE repository should grow to include not just data, but the code
required to run experiments over that data. We look forward to the day
when it is routine for PROMISE submissions to come not just with
supporting data but also with a full executable version of the
experimental rig used in the paper. This would simplify the goal of 
letting other researchers repeat, and maybe even improve, the experimental results of others.

In the paper, we have used OURMINE to reproduce an important result:
\bi
\item It is best to use local data to build local
defect predictors;
\item
However, if imported data is selected using a 
{\em relevancy filtering}, then....
\item
...  imported data can build defect
predictors that function nearly as well as those built from
local data.
\ei
This means that repositories like PROMISE are actually an important
source of data for industrial applications.

We prefer OURMINE to other tools. For example, four features of \fig{example1}
and \fig{an2} are worthy of mention:
\be
\item
OURMINE is very succinct, a few lines can describe even
 complex experiments.
\item
OURMINE's experimental descriptions are  complete. There is nothing hidden 
in \fig{an2}; it is not the pseudocode of the our experiments,
it \underline{is} the experiment.
\item
OURMINE code is executable and can be  executed by
other researchers directly.
\item
Lastly, 
the execution environment of OURMINE is readily available.
Many machines already have the support tools required for OURMINE. For example,  
we have run OURMINE on
Linux, Mac, and Windows machines (with Cygwin installed). 
\ee

Like Ritthol et al., we doubt that the standard interfaces of tools
like WEKA, etc, are adequate for representing the space of possible
experiments. Impressive visual programming environments are not the
answer: their sophistication can either distract or discourage
novice data miners from extensive modification and experimentation.
Also, we find that the functionality of the visual environment can
be achieved with little BASH and GAWK scripts, with a fraction
of the development effort and a greatly increased chance that novices
will modify the environment.

OURMINE is  hence a candidate format for sharing descriptions
of experiments. The PROMISE community
might find this format unacceptable but discussions about the drawbacks (or strengths)
of OURMINE would help evolve not just OURMINE, but also the discussion on how
to represent data mining experiments for software engineering.

\bibliographystyle{plain}
\bibliography{refs}

\appendix
\section*{Installing OURMINE}
OURMINE is an open source tool licenses under GPL 3.0. It can
be downloaded from
\url{http://unbox.org/wisp/trunk/our/INSTALL}.

As OURMINE is a command-line tool, the system requirements are insignificant. 
However, there are a few things that are necessary before installing OURMINE.
\bi
\item A Unix-based platform. OURMINE is designed to work in the BASH shell, and will not operate on a Windows system.
If no other platform is available, a BASH emulator like Cygwin will need to be installed before using OURMINE. 
Users running any Linux distribution, BSD, or Mac OS X can run OURMINE natively. 
\item The Java Runtime Environment. Most computers will already have this installed. The ability to run Java programs 
is required for the WEKA learners.
\item The GAWK Programming Language. Many of the scripts within OURMINE are written using GAWK. Any up-to-date Linux
system will already have GAWK installed. Cygwin or Mac OS X users will need to install it themselves. 
\ei
To install OURMINE, follow these instructions:
\bi
\item Go to a temporary directory
\item wget -q -O INSTALL \newline http://unbox.org/wisp/trunk/our/INSTALL
\item bash INSTALL
\ei
OURMINE is installed to the \$HOME/opt directory by default. To run OURMINE, simply move into that directory 
and type {\em bash our minerc}. This will launch the OURMINE shell. 
OURMINE has several demos included to familiarize you with the built-in functionality. These demos may be run by typing 
{\em demoX} into the command prompt, where {\em X} is a number between 3 and 19. To look at the source code for that demo, type
{\em show demoX}. It is highly recommended that new users run each demo and take a close look at its source code. 
This {\em show} command works for any function in OURMINE, not just the demos. 


%\begin{thebibliography}{10}


%\end{thebibliography}


\end{document}

