
\newtheorem{future}{Future work}

\section{Validity \& Future Work}

\emph{Internal Validity}: Apart from joining the PROMISE data sets (for defect counts) with the Helix data sets
(for the activity measures), we did not pre-process the datasets in any way. This was done to enable 
replication of our results. 

\emph{Construct Validity}: We have made the case above that the measures of \fig{metrics} reflect the ``activity''
of different classes, that is, how often a developer will modify or extend the services of a class as an expression of the attractiveness of this class for the developer's design choices.
This case has not been tested here. Hence:

\begin{future}
Analyze participant observation of developers to determine what classes they inspect as part of their workflow. 
\end{future}


\emph{External Validity}: Our use of cross-validation means that all the results reported above come from the application of our models to data not seen during training. This gives us some confidence that these results will
hold for future data sets.

As to our selection of data sets, the material used in this study represents real-world use, collected from
real-world  projects. Measured in terms of number of data sets, this paper is one of the largest defect
prediction studies that we are aware of. Nevertheless, there is a clear bias in our sample: open-source 
Java systems. Hence:

\begin{future}
Test the validity of our conclusions to close-sourced, non-object-oriented, and non-Java projects.
\end{future}

\emph{Conclusion Validity}: We take great care to \emph{only} state our conclusions in terms of areas under a 
\%LOC-vs-recall curve. For the purpose
of finding the most defects after inspecting the fewest lines of code (\ie, the inspection optimization criterion
proposed by Arisholm \& Briand~\cite{arisholm06}), the ``activity'' measures of \fig{metrics} %are nearly perfect
exhibit an excellent performance
 (median results, within 96\% of the optimum).

While area under a \%LOC-vs-recall is an interesting measure, it is not the only seen in the literature. Hence:

\begin{future}
Explore the value of activity for other evaluation criteria.
\end{future}

Those other criteria include:

\bi
\item 
Counting the number of files inspected, rather than the total LOC, as done, for example, by Weyuker and 
Ostrand~\cite{ostrand04,weyeuker08},
\item 
Precision, as advocated, for example, by Zhang \& Zhang~\cite{zhang07} (but depreciated by Menzies 
\etal~\cite{me07e}),
\item
Area under the curve of the \emph{pd}-vs-\emph{pf} curves, as used, for example, by Lessmann 
\etal~\cite{lessmann09}.
\ei

