\section{Threats to validity}
In prior work we have explored a range of data mining methods
for defect prediction and found that
classifiers based on Bayes theorem work best for the 
\tbl{data1} data~\cite{me07b}.
Since that study, we have tried to find  better data mining
algorithms for defect prediction. To date, we have failed. 
%Our
%recent (as yet, unpublished) experiments have found no additional
%statistically significant improvement from the application of the
%following data mining methods: logistic regression; average
%one-dependence estimators; under- or
%over-sampling~\cite{drummond03}, random forests,
%RIPPER~\cite{cohen95r}, decision tree learning with 
%J48~\cite{quinlan92}, rule learning with OneR~\cite{holte93}
%and bagging~\cite{brieman96}. Only boosting~\cite{FreSch97} on
%discretized data offers a statistically better result than a Bayes
%classifier. However, we cannot recommend boosting: the median
%improvement is quite negligible and  boosting is orders of
%magnitude slower than our simple Bayes classifier.
Other researchers have also failed to improve our results. For example, Lessmann et al. 
investigate the statistical difference of the results between 19 learners, including naive Bayes, on the NASA projects~\cite{lessmann09}. \fig{lessmann} shows that the simple Bayesian method discussed above ties in
first place along with 15 other methods.


\begin{figure}
\begin{center}
\includegraphics[width=3in]{lessmann.pdf}
\end{center}
\caption{x-axis shows the rank of the data miners given in the y-axis. All the methods
whose top ranks are 4 to 12 are statistically insignificantly different.
From~\cite{lessmann09}.
}\label{fig:lessmann}
\end{figure}

Our analysis do not involve a value-based approach. We do not consider that fixing defects in a module may be more critical than fixing defects in another one. A reason is that our data sources do not consistently include 
necessary data (i.e. the severity level of defects). If and when data are available, same kind of analysis should be repeated considering this issue. 

Another aspect is that, descriptions of software modules {\em only} in terms of static
code attributes can overlook some important aspects of software
including
the type of application
domain; the skill level of the individual
programmers involved in system development;
 contractor development practices;
the variation in measurement practices;
and the validation of the measurements and instruments used
to collect the data.
For this reason some researchers augment, or replace
static code measures with other information such as the history
of past faults or changes to code or number of
developers who have worked on the code~\cite{graves00}. Yet again, we have used all available data in our data sources, which are the static code features of the projects.
 
The external validity of generalizing from NASA examples
has been discussed elsewhere~\cite{me07b}. In summary, 
NASA uses 
contractors who are contractually obliged to demonstrate
their understanding and usage of current industrial best
practices. These contractors service many other industries;
for example, Rockwell-Collins builds systems for many government and
commercial organizations.  For these reasons, other noted researchers
such as Basili, Zelkowitz, et al.~\cite{basili02} have argued that
conclusions from NASA data are relevant to the general software
engineering industry.

Nevertheless, it is always wise to test claims of external validity. Hence:
\bi
\item
The SOFTLAB data were, initially,  in reserve.
 Our first three experiments
are based solely on the aerospace applications found in the NASA data.
\item
Our last experiment checked if the  SOFTLAB data exhibit the same pattern as the 
NASA data.
\ei

SOFTLAB data sets were deliberately chosen to be as
far removed as possible from American aerospace software applications
(i.e. Turkish home appliances controller software).
Please note that this software was developed via methods that are both
culturally and organizationally different to NASA aerospace software. Turkish domestic appliances company software are developed by a small team of 2-3 people. The development is carried out in an ad-hoc, informal way rather than formal, process oriented approach in NASA. Furthermore, the company is a profit and revenue driven commercial organization, whereas NASA is a cost driven government entity. 


