0;136;0cAfter having experimented with {\W}2 we were impressed with its execution speed and generally pleased with the reductions stemming from its recommendations. That being said, {\W}2 is not without its problems. There are some noticeable issues we ran into that we feel require being explored in more detail.

\subsection{Result Consistency}

As discussed briefly above, we consider the reliability of {\W}2's results its only real shortcoming at this point. To apply a quality optimizer's recommendations and feel confident of this decision, we have to be certain that the software is capable of consistently providing us with the best possible results for the project at hand. In order to attain a certain level of consistency within our results we believe there might be several possible avenues worth exploring further.

\subsection{Larger Datasets}

One possible way of increasing the reliability of our results is using much larger datasets. The reasoning here being that as we increase the amount of historical instances available, we increase the number of projects that highly overlap our test case. It follows that if the project metrics present in the learning data are all closer to our test instance, the best set should be better and the results we obtain in the end should be more precise. The preliminary tests we ran with large datasets were used only to record runtimes. We took the largest effort estimation dataset available, namely ``china'', and copied and pasted its data back into itself until we ended up with a series of files containing between 2 and 512 times the amount of historical instances in the original dataset. Simply replicating the data within a given dataset is assumed to have some effect on the results, however since we are testing for run times we are not concerned with results at this point. Figure 4 graphically demonstrates the linear behavior. Testing whether or not there is an improvement in results with larger datasets remains to be explored.

\subsection{Size of Best and Rest Sets}

The only method we found that allowed us to affect the consistency of the recommendations was to modify {\W}2's K1 and K2 parameters. K1 represents the number of instances in the best set, while K2 represents the number of instances in the rest set, therefore the sum of these two values is the number of instances that most overlap the test project. All of the previous results obtained from {\W}2 used a value of 20 for K1+K2, where K1=5, K2=15 [2]. What we found is that increasing the size of the best and rest sets has a very positive effect on the stability of {\W}2's results. Not only did both the median and spread values became consistent among separate instances of 30 runs, but the reduction values themselves increased dramatically. During our trials we used the values 10 and 30 for K1 and K2 respectively. Figure 5 contains the results from running nasa93/ground with 100 trials 8 times. Half of the trials were conducted with K1=5, K2=15, these are the first 3 columns. The last 3 columns represent the other half of the trials that were conducted with K1=10, K2=30. Figure 5 clearly shows that not only did the reductions increase in every individual utility metric, but consistency from one set of 100 runs to another improved as well. A curious observation is that the new K values prevent the minimum median and spread reductions from falling as dramatically as they would otherwise. While no objective run time measurements were taken, no performance decrease was noticed with this minor change in place. Despite this, we recommend run time testing with modified K1, K2 values be a part of any future work focusing around {\W}2.

\subsection{Further Considerations on Best/Rest}

Given the observations outlined in section 5.3, an interesting question arises with respect to changing the sizes of the best and rest sets. We believe it is worth investigating whether or not there exists a pair of K1 and K2 that maximizes the consistency and the magnitude of {\W}2's reduction values. Furthermore, if it turned out that such a pair of values exists, the relationship between dataset and these values should be explored. Such a relationship, if known, would allow us to pre-process the data and dynamically set K1 and K2 at runtime such that consistency and metric reduction are both maximized per dataset.

There is also the possibility that the improvements we are seeing in our recommendations are in fact the result of deeper issues within the system and not indicative of an overall positive effect. Further experimentation is required to determine which is the case.

\subsection{Effects of Training Data Quality}

One important consideration we have to keep in mind is that any results obtained can only be as trustworthy and reliable as the historical data used in training. Having said that, there are several areas in which the data we use for learning can be lacking in. Examples of what we consider to be poor quality data include very short datasets as well as datasets containing utility metrics with very limited ranges of values.

\subsection{Very Short Datasets}

We observed the effects of both types of data described above during our exploration of defect prediction datasets. Very short datasets, containing a handful of historical instances, are not only not well suited to our goal of extracting information, but also result in {\W}2 terminating with a ``divide by 0'' error a majority of the time. Switching to a dataset with regular length, we observe the ``divide by 0'' errors disappear. To perform our tests on the effects of short datasets, we took a working, effort estimation dataset and removed all but the first 4 historical instances. We alternated between using the original data and the shortened copy all while keeping the test project file constant. The main conclusion we can draw from using very short datasets is that {\W}2 could benefit greatly from the addition of some rudimentary error prevention. Ideally, {\W}2 would never accept running with a dataset shorter than an established minimum length.

\input{fig4.tex}

\subsection{Limited Utility Metric Ranges}

Another aspect of data quality became clear to us as we tried using different defect prediction datasets. What we found is that for a dataset containing a utility metric with a range of only a handful of values, the median and spread reductions will almost always be 0. We first noticed this situation with the ``Ant'' dataset. As we used the single utility metric ``bug'', we consistently obtained no reductions in median or spread. Attempting to understand why our results were so reliably bad, we took a look at the raw data and noticed our utility metric had a range from 0 to 3 containing a mere 4 values. To ascertain whether or not this was the cause of our results, we applied the same technique described in section 5.6. We took a working, effort estimation dataset, and modified its contents to the point at which the range of its utility metric contained only 5 values. We then ran {\W}2 against the original and the modified datasets and observed that over any number of runs the reduction percentages consistently stay at 0.

The limited range exhibited by the defect prediction data's utility metric is either an indication of poor quality or the fact that the data is unsuited to reducing the metric we chose. In this case we considered the data to be of poor quality. The defect datasets contain what appear to be historical instances of projects with very low numbers of bugs. Whereas anywhere else bug-free software is a good thing, the absence of any instances with significant numbers of bugs makes it impossible for {\W}2 to gauge which attributes could lead to improvements. It follows that for the purpose of minimizing the number of bugs in software, several of the defect prediction datasets we tried contain poor quality data. If the utility metric is already reduced as far as it will go, there is nothing left to optimize.

This is another problem area that could be taken care of by a level of error protection built into the system. {\W}2 could pre-process the historical data and make sure the utility metrics have a wide enough range of values so as to avoid running into this issue.

\subsection{Effects of Training Data Type}

What do we mean when we use the term ``type'' while discussing datasets? In this context we apply the term ``type'' to those characteristics which we observe due to what the data in question represents. One clear example of such a characteristic is the number of attributes per instance contained in a given dataset. Such wider datasets could impact the performance of the system in unexpected ways. There may also be cases in which the ranges exhibited by the values of whatever utility metrics are used depend on what the data represents and not its quality, as was discussed in section 5.7.

Considering the above points, we believe that, besides its quality, it may be possible for the type of data used to influence how well the system performs and that this should be a topic for research in future work involving {\W}2.
