While working with {\W}2, one serious issue that would hinder the usability of this software outside of a laboratory setting, becomes immediately apparent. The existing {\W}2 codebase is plagued by inconsistencies in the recommendations it suggests. The instability of {\W}2's results shows up both when performing individual runs as well as when compiling statistical results from multiple runs. Consistency issues persist across different datasets as well. We believe the combination of inconsistency across  individual runs and inconsistency present across statistical results from multiple trials can make it difficult to correctly interpret results.

\input{fig2.tex}
\input{fig3.tex}

\subsection{Single and Multiple Trial Inconsistancy}

When performing single runs, {\W}2 comes up with, sometimes wildly, varying results from run to run. A trial with median and spread reductions of well over 50\% will be followed by a trial with negative reductions. Whenever {\W}2's recommendations lead to a negative reduction in utility metrics, that is an increase in utility metrics, we consider these recommendations faulty. Such trials are uncommon, however they do occur. There also does not appear to be any discernible pattern as to when {\W}2 will return a faulty recommendation. The same thing happens after running {\W}2 a number of times, in our case 30, and noting the medians of the median reduction and spread reduction sets collected from each run, the same instability becomes noticed upon repeating the same number of runs. Figure 3 demonstrates this by listing the results from 3 runs of 1, 30, and 60 trials. The values for the runs of 30 and 60 trials are the medians of the values collected from the trials.

\subsection{Increasing Number of Trials}

While there is nothing one can do about instability from one trial to another without changing the software, over many runs statistical analysis can bring results to the surface. At first we assumed it would be possible to remove the consistency issues by running {\W}2 a large enough number of times and looking at the medians of the median and spread reduction sets collected from each run. To find out if this would work we increased the number of trials incrementally from 30 to 100 runs without noticing any improvement in result consistency. While it may be possible that all one needs to do to avoid this lack of consistency is run {\W}2 a large enough number of times, we could not find it in our tests. If there is a statistical limit the results tend towards, more than 100 trials would be required to find it. We did learn that the number of runs resulting in faulty recommendations, that is recommendations that lead to increased utility metrics, increases as more and more trials are performed. 

\subsection{Inconsistency Across Datasets}

The issue of inconsistency remains present while using {\W}2, regardless of dataset choice. We cannot state that the dataset used has no influence on consistency, only that inconsistency was observed among all datasets tested. Future research applying different types of historical data will show if and how {\W}2's performance is affected.


