COSEEKMO is an open-source tool designed for performing effort estimation experiments.
It consists of several effort estimation methods that are all based on COCOMO-style data sets.
Although these methods are used on COCOMO-style data sets, COSEEKMO does not have such restrictions
and can incorporate any method. The goal of COSEEKMO is to allow the user to compare different effort estimation methods and
obtain benchmarks for new methods comparing to existing methods including the well-known methods in the literature and industry.
Therefore, COSEEKMO can be expanded using new methods. This manual explains how to use COSEEKMO and how to
add, remove, or modify a certain method.

\section{Obtaining the Tool}
\label{a:obtain}

COSEEKMO can be found at \url{http://unbox.org/wisp/tags/coseekmo/2.0}.
All directories and their files must be downloaded for COSEEKMO to work properly.
COSEEKMO is a command line tool that can be used in the Terminal of any Linux distribution such as
Ubuntu (available for free at \url{http://www.ubuntu.com}).
Although COSEEKMO was designed with Linux as the operating system in mind, Mac users can modify
it to run in X11 or Terminal. Also, Windows users can use it without any modification under Cygwin
(available for free at \url{http://cygwin.com}).

Regardless of what operating system is used, certain packages must be installed. These packages are:
\begin{itemize}
\item bash: the required shell for COSEEKMO,
\item g++: the compiler for C++ language,
\item gawk: the interpreter for the gawk language, and
\item java: the interpreter for the java language. 
Java is used to run Weka and as of the date that this document is written, Weka requires Sun Java Runtime Environment
(available for free at \url{http://www.java.com}) rather than the Java in combination with Linux or Gnome.
\end{itemize}

\section{Directory Structure}

COSEEKMO is divided into several parts each represented by a directory.
The major files of COSEEKMO are stored in the {\em bin} directory.
There are certain configuration files that are included in the {\em config} directory 
and are used by COSEEKMO in numeric estimations. Directories {\em data} and {\em weka}
contain the data sets and the Weka data mining software (available for free at 
\url{http://www.cs.waikato.ac.nz/ml/weka}) respectively. Directory {\em eg} contains examples with certain
methods and their combinations. Directory {\em evaluation} contains the evaluation tools used in COSEEKMO and specified in the {\em start} script.

Each method has its own directory, which includes the files necessary to run that method individually.
As explained in Sections \ref{a:inputFormat} and \ref{a:outputFormat}, each method has to
conform to certain input and output formats.
Since modifying an already existing method might not be possible, scripts are
provided in the {\em methods} directory to modify the input and output accordingly.
These scripts run the methods and are the only way that COSEEKMO uses to communicate to the methods.
If a new method is added, it is strongly suggested that the existing scripts are used
as templates to create new scripts for such a method. However, not all methods require scripts and some
(such as column pruners) can be directly added to the COSEEKMO's main script, {\em generate}.

After the experiment is finished, the results are stored into {\em results} directory.
This directory contains graphing scripts as well.

\section{Input Format}
\label{a:inputFormat}

Each method in COSEEKMO may have different input formats and accept different parameters.
However, before such a method can be used in COSEEKMO, a uniform input format must be adopted.
In order to facilitate the adoption of such formats, scripts can be used to convert
the input provided by COSEEKMO to the input format require by the method. This eliminates
the need for changes in methods' source codes and the need for recompilation.
COSEEKMO provides two input files for training and testing a model as well as a string
called {\em Prefix} that contains the structure of the output requested from the
method and can be filled in with the required information provided by the method.
This is further explained in Section \ref{a:outputFormat}.

The train and test files are generated by COSEEKMO, enabling it to compare methods
on the same input. Although each method's use for the input files may be different from the others,
in a specific run all corresponding input files (whether the train or the test file) are the same for 
all methods. Moreover, every input file is an ARFF (Attribute-Relation File Format) file.
As a part of the experiments, COSEEKMO handles the conversion of the COCOMO cost drivers from the ordinal values 
to numeric values used as effort multipliers. Hence, all input files contain these numeric values before they can
be used by the methods.

\section{Output Format}
\label{a:outputFormat}

Since COSEEKMO is used to compare different methods, the output of all methods must conform to what COSEEKMO
can use in its evaluation methods. The format of the output is passed as an input parameter to each method.
The output is in CSV (Comma-Separated Value) format and each field can be completed according to the method
used and the information, such as the name of the method, that is necessary to distinguish that method from other methods.

This output format can be modified if necessary as long as all methods and evaluation tools reflect this
change accordingly. However, the current output format contains the necessary fields for a complete comparison of methods.
These fields, in the current order are:
\begin{itemize}
\item a number representing the current run, which is incremented after each run in finished,
\item a string representing the name of the data set (or its subset) used to generate the input files for training and testing,
\item a string representing the name of the method used to create the train file,
\item a string representing the name of the numeric estimation method (currently either precise or proximal),
\item a string representing the name of the column pruning method used as the pre-processor (or {\em None} if none used),
\item a string representing the name of the row pruning method used as the pre-processor (or {\em None} if none used),
\item a number representing the number of attributes used from the train file by the method (it may be smaller than the
normal value only if a column pruning method was used as a pre-processor),
\item a number representing the number of instances used from the train file by the method (it may be smaller than the
normal value only if a row pruning method was used as a pre-processor),
\item a string representing the name of the learner used (or {\em None} if none used),
\item a number representing the predicted value of cost for a test instance, and
\item a number representing the actual value of cost for a test instance.
\end{itemize}

This ordering represents a combination of zero or one column pruning methods, zero or one row pruning methods, and zero or one learners.
The method used can be any one of these three or a combination of these three.

Since column pruning can affect the number of attributes in the train file, the number
of attributes in the test file is adjusted (to the number of attributes in the train file)
accordingly in order to test the trained model.
However, row pruning only affects the number of instances in the train file when training a model. 
As a result, each and every test instance can be used when testing the trained model. 
Hence, the number of test instances is always a constant value. 
Clearly, it is redundant to include these two numbers (number of attributes and number of instances
of the test file) in the fields of the string above.

\section{Adding a Method}
\label{a:add}

In order to add a method, a directory can be created in the COSEEKMO tool and all the method's files can be
added to that directory. As explained before, the method needs to conform to the input and output formats of
COSEEKMO. Therefore, the user can add a script to the {\em methods} directory. This script will be responsible
for communicating between COSEEKMO and the method's files stored in its directory.
This way, COSEEKMO does not have to know anything about how the method works. The script handles everything
the method does and only submits the results to COSEEKMO.
Therefore, this script is responsible for converting the input format of COSEEKMO to the input format
required by the method and also converting the output of the method to the
output format that COSEEKMO requires.

There are three different types of methods that can be added to COSEEKMO.
\begin{itemize}
\item The column pruners are the first group. COSEEKMO calls them before calling a row pruner or a learner.
They can be added to the {\em ColumnPrunerMethods} category in the {\em start} script.
\item Another group is learners and row pruners that can operate on the results of a column-pruned train and test file.
This group is called automatically after a column pruner has finished processing.
They can be added to the {\em NormalMethods} category in the {\em start} script.
\item Finally, there are methods that require no column or row pruning.
They can be added to the {\em SpecialMethods} category in the {\em start} script.
The benefits of this category are further explained in Section \ref{a:combine}.
\end{itemize}

In special cases, a method can be added directly to the main script of COSEEKMO, {\em generate}.

\section{Enabling a Method}

A method must be added to one of the three categories of {\em ColumnPrunerMethods}, {\em NormalMethods}, or {\em SpecialMethods}
in the {\em start} script in order to be enabled. 

\section{Removing a Method}

Any method can be removed from COSEEKMO by removing its directory and files as well as the scripts added to the {\em methods} directory.
It must also be disabled by removing it from any of the three categories of {\em ColumnPrunerMethods}, {\em NormalMethods}, or {\em SpecialMethods}
in the {\em start} script.

\section{Disabling a Method}

A method must be removed from any of the three categories of {\em ColumnPrunerMethods}, {\em NormalMethods}, or {\em SpecialMethods}
in the {\em start} script in order to be disabled.

\section{Modifying a Method}

Since COSEEKMO only requires certain input and output formats, modifying a method has no effect on COSEEKMO's functionality as long
as the script created for the method satisfies the input and output formats. Therefore, the user can modify a method at any time.

\section{Combination of Methods}
\label{a:combine}

COSEEKMO allows for combining different methods. Therefore, it is possible to create various combinations of methods in COSEEKMO.
By default, COSEEKMO combines all column pruners with all the normal methods specified in the {\em NormalMethods} category.
As briefly explained in Section \ref{a:add}, a method in the {\em NormalMethods} category is either a learner, a row pruner, or both.
Therefore, the combination of a column pruner with zero or one row pruners and zero or one learners can be done automatically by adding
a method that combines zero or one row pruners with zero or one learners.

In order to make the column pruners optional, the user can add a method to the {\em SpecialMethods} category.
This method can be a combination of zero or one row pruners with zero or one learners. Therefore, all possible combinations of
zero or one column pruners, zero or one row pruners, and zero or one learners is supported in COSEEKMO.

The {\em SpecialMethods} category gives limitless power to the user by allowing the user to 
combine more than one of each type of the methods and obtain a combination that can potentially 
have several layers of pre-processors and learners. This allows the user to explore countless combinations,
including any ordering of column pruners, row pruners, and learners.

\section{Running the Tool}

After obtaining all the files and directories of COSEEKMO (described in Section \ref{a:obtain}),
the user can run it by running the script called {\em start} in the main directory of COSEEKMO.
This script first prepares the tool for running the methods.
It then runs all the methods specified in the three categories of {\em ColumnPrunerMethods}, {\em NormalMethods},
and {\em SpecialMethods} and creates a log of the results. Finally, it runs the specified evaluation method
and stores the results for each data set or subset in a separate file.

There are three example scripts that can be used as well. They are located in the {\em eg} directory. 
The user can run these example scripts instead of the {\em start} script in the main directory.
