
%%% Local Variables: 
%%% mode: latex
%%% TeX-master: t
%%% End: 
\section{Experiments}

\begin{figure}[!t]
\begin{center}
{\scriptsize\begin{tabular}{l@{~}l@{~}l@{~}r@{~}c} 
\multicolumn{4}{c}{~}& \multicolumn{1}{c}{2nd quartile, median }\\
data &rank & treatment & median $C'_2$ & \multicolumn{1}{c}{and 3rd quartile} \\\hline
all& 1 &      which2&  70.9&\boxplot{   0.0}{  52.5}{  70.9}{27}{ 100.2}\\
& 1 &     micro20&  67.0&\boxplot{   0.0}{  50.0}{  67.0}{25.9}{ 100.0}\\
&2 &    manualUp&  61.1&\boxplot{   0.0}{  52.0}{  61.1}{16.4}{  99.3}\\
&3 &       Bayes&  56.8&\boxplot{   0.0}{  47.5}{  56.8}{15.6}{ 100.2}\\
&4 &  manualDown&  49.5&\boxplot{   0.0}{  43.8}{  49.5}{14.9}{ 100.5}\\
&5 &      which4&  49.2&\boxplot{   0.0}{  30.1}{  49.2}{29.6}{ 100.0}\\
&6 &      which8&  31.2&\boxplot{   0.0}{   1.7}{  31.2}{40.9}{ 100.0}\\
&6 &         j48&  27.6&\boxplot{   0.0}{  16.9}{  27.6}{26.3}{ 100.1}\\
&7 &        jRip&  21.9&\boxplot{   0.0}{  10.6}{  21.9}{26.3}{ 100.2}\\\hline
&& & & \multicolumn{1}{c}{50\%} 
\end{tabular}}
\end{center}
\caption{Results from all data sets of \fig{data}, combined from 10 repeats of a 3-way cross-val,
sorted by {\em median} $C'_2$. 
The $rank$ values in column 1 were generated in the same manner as \fig{bands}.
}\label{fig:allr}
\end{figure}

\PARstart{F}{igure} \ref{fig:allr} shows results from experimental runs on the \fig{data}. Each run
randomized the order of the data ten times, then performed a N=3-way cross-val study
(N=3 was used since some of our data sets were quite small). For each part of the cross-val study,
pd-vs-effort curves were generated using:
\bi
\item {\em Manual methods}: manualUp and manualDown;
\item {\em Using standard data miners}: the j48 decision tree learner, the jRip rule learner, and our previously
recommended  Bayes method. For more details on these leaners, see the Appendix I,II, and III.
Note that these standard miners including methods that we have advocated in prior publications \cite{%
jiang07,%
me00p,%
me02g,%
me02k,%
me03q,%
me04c,%
me05d,%
me07b,%
me07e%
}. 
\item Two versions of {\em WHICH}: 
WHICH2 discretizes continous ranges with a log filter, then a divides the numerics into two equal width bins;
micro20 is WHICH2, plus the micro-sampling strategy discussed in \sect{micro}. For more details on WHICH,
see Appendix IV.
\ei
To be accurate, numerous versions of WHICH were explored, each  with different discretization policies. We only
report
WHICH2 and micro20 since they were  always on the upper envelope of the results. 


%\begin{figure}
%\begin{center}
%{\scriptsize
%\begin{tabular}{l|lrrrr}
%rank & treatment & wins & ties & losses  \\\hline
%1 &      which2&    1&   7&    0\\
%1 &     micro20&    1&   7&    0\\
%2 &    manualUp&    0&   6&    2\\
%3 &       Bayes&    0&   5&    3\\
%4 &  manualDown&    0&   4&    4\\
%5 &      which4&    0&   3&    5\\
%6 &         j48&    1&   1&    6\\
%6 &      which8&    2&   0&    6\\
%7 &        jRip&    1&   0&    7\\
%\end{tabular}}
%\end{center}
%\caption{Mann-Whitney (99\%) analysis of the \fig{allr} results.  Each row shows
%a comparison of one treatment agains the other eight.
%The rankings, shown on the left-hand-side,
%group together any treatments with the same ties+losses.
%}\label{fig:allrmann}
%\end{figure}

\subsection{Overall Results}
\fig{allr} shows the results for {\em all} the  data sets of \fig{data}, combined.
Several aspects of this figure are noteworthy.
\bi
\item
In a result consistent with our prior publications~\cite{me07b},  our Bayes
classifier
out-performs other standard data miners (j48 and jRip).
\item
In a result consistent with Koru et.ak's logarithmic defect hypothesis,
manualUp defeats manualDown. 
\item
Also, in a result consistent with the limited information hypothesis, micro20 has the same rank as WHICH2
(i.e. their performance is statistically indisguisable). Recall that, under micro20-sampling, learning is
performed using 40 examples evenly mixed of defective and non-defective modules.
\item
Further, in a result consistent with 
the core assumption of this paper (that is useful to make $C_1\approx C_2$), varaints of
WHICH out-performs
all other methods.
That is, unlike our prior results,  all the learners in this study do {\em not} 
hover around the same  performance ceiling.
At least for the purposes of $application_x$, the WHICH learners have a demonstrably superior performance.
\item
These results calls into question the relevance of numerous prior results
for addressing $application_x$
(including many papers written by the authors
\cite{%
jiang07,%
me00p,%
me02g,%
me02k,%
me03q,%
me04c,%
me05d,%
me07b,%
me07e%
}). In \fig{allr}, standard data miners are defeted by a manual method (manualUp). 
The size of the defeat is very large: median values of 61.1\% to 27.6\% from manualUp to j48.
One very sobering result
in \fig{allr} is that two widely used methods (j48 and jRip) are defeated by inspection methods
with questionable
value. 
\ei
%data mining technology. performs very well, defeating 

\subsection{Internal Validity}

\fig{allr} showed combined results from all the data sets
of \fig{data}. From it, we concluded that
WHICH is preferred to other methods for $applciation_x$.

Figures \ref{fig:bands1},
\ref{fig:bands2}, and
\ref{fig:bands3} checks the internal validity of the this conclusion
by looking at each data set in isolation. 
The results divide into three patterns:
\bi
\item In the eight data sets of pattern \#1 (shown in \fig{bands1}),
WHICH2 has {\em both} the highest median $C'_2$ performance 
{\em and} is found in the top rank by the Mann-Whitney statistical analysis.
\item
In the two data set of pattern \#2 (shown in \fig{bands2}),
WHICH2 does not score the highest median performance, but still
is found in the top-rank.
\item
In the one data of pattern \#3 (shown in \fig{bands3}),
WHICH2 is soundly defeated by manual methods (manualUp).
However, in this case, the WHICH variant micro20 falls into the second rank
\ei
In summary, when looking at each data set in isolation, WHICH2 performs very well
in $\frac{10}{11}$ of the data sets. Hence, we say that the general conclusion
of the last section
almost always works for specific data sets.


\begin{figure}
\begin{center}
{\scriptsize\begin{tabular}{l@{~}l@{~}l@{~}r@{~}c} 
\multicolumn{4}{c}{~}& \multicolumn{1}{c}{2nd quartile, median }\\
data &rank & treatment & median $C'_2$ & \multicolumn{1}{c}{and 3rd quartile} \\\hline
ar4 &1 &      which2&  58.6&\boxplot{  33.6}{  49.6}{  58.6}{16.4}{  78.6}\\
& 1 &  manualDown&  56.5&\boxplot{  42.5}{  45.2}{  56.5}{18.5}{  84.2}\\
& 1 &       Bayes&  56.2&\boxplot{  33.6}{  47.7}{  56.2}{18.2}{  76.6}\\
& 1 &    manualUp&  56.2&\boxplot{  32.0}{  45.7}{  56.2}{16.9}{  70.4}\\
& 1 &      which4&  49.3&\boxplot{  11.6}{  36.7}{  49.3}{24.3}{  71.9}\\
& 1 &        jRip&  47.3&\boxplot{   0.0}{  34.0}{  47.3}{21.8}{  87.7}\\
&2 &     micro20&  49.2&\boxplot{   0.1}{  38.5}{  49.2}{17.9}{  78.6}\\
&3 &      which8&  42.6&\boxplot{   0.0}{  31.3}{  42.6}{15.3}{  66.1}\\
&3 &         j48&  38.8&\boxplot{   9.2}{  21.2}{  38.8}{31.2}{  67.2}\\\hline
cm1&1 &      which2&  68.1&\boxplot{   0.0}{  57.4}{  68.1}{14.1}{  81.5}\\
& 1 &     micro20&  64.7&\boxplot{   0.0}{  48.9}{  64.7}{21}{  89.1}\\
& 1 &    manualUp&  59.8&\boxplot{  48.3}{  57.4}{  59.8}{7.9}{  71.5}\\
&2 &       Bayes&  52.1&\boxplot{  36.2}{  46.0}{  52.1}{13.1}{  69.2}\\
&3 &  manualDown&  47.6&\boxplot{  33.6}{  40.3}{  47.6}{9}{  60.2}\\
&4 &      which8&  11.4&\boxplot{   0.0}{   0.0}{  11.4}{26.2}{  35.6}\\
& 4 &        jRip&   5.8&\boxplot{   0.0}{   0.0}{   5.8}{11.5}{  24.1}\\
& 4 &         j48&   0.1&\boxplot{   0.0}{   0.0}{   0.1}{12.9}{  33.3}\\
& 4 &      which4&   0.0&\boxplot{   0.0}{   0.0}{   0.0}{41.2}{  69.0}\\\hline
kc1&1 &      which2&  76.0&\boxplot{  71.4}{  73.8}{  76.0}{4.2}{  81.8}\\
&1  &     micro20&  75.1&\boxplot{   1.9}{  70.3}{  75.1}{5.9}{  80.1}\\
&2 &    manualUp&  67.6&\boxplot{  64.5}{  65.8}{  67.6}{3.1}{  70.0}\\
&3 &       Bayes&  61.9&\boxplot{  54.9}{  60.2}{  61.9}{2.8}{  67.7}\\
&4 &      which4&  52.9&\boxplot{   0.0}{  49.8}{  52.9}{5.4}{  60.5}\\
&5 &  manualDown&  43.3&\boxplot{  39.7}{  42.2}{  43.3}{3}{  47.7}\\
&6 &         j48&  27.8&\boxplot{  11.6}{  20.5}{  27.8}{11.2}{  40.1}\\
&7 &        jRip&  21.3&\boxplot{  10.2}{  17.3}{  21.3}{7.9}{  32.4}\\
&8 &      which8&   0.0&\boxplot{   0.0}{   0.0}{   0.0}{2}{  33.9}\\\hline
kc2&1 &      which2&  81.6&\boxplot{  65.6}{  76.0}{  81.6}{8.6}{  88.5}\\
&2 &     micro20&  74.8&\boxplot{  61.1}{  71.6}{  74.8}{7.8}{  88.4}\\
&3 &    manualUp&  69.3&\boxplot{  57.9}{  65.4}{  69.3}{5.6}{  76.6}\\
&4 &      which4&  59.4&\boxplot{  43.1}{  52.5}{  59.4}{14.3}{  79.6}\\
&4 &       Bayes&  58.7&\boxplot{  47.0}{  54.8}{  58.7}{6.2}{  69.4}\\
&5  &  manualDown&  46.1&\boxplot{  37.9}{  43.1}{  46.1}{9.2}{  62.4}\\
&5  &        jRip&  42.2&\boxplot{  22.2}{  36.0}{  42.2}{13.5}{  65.2}\\
&6 &      which8&  41.2&\boxplot{  26.3}{  36.5}{  41.2}{11.1}{  56.5}\\
&6 &         j48&  41.2&\boxplot{  26.0}{  36.1}{  41.2}{9.8}{  59.8}\\\hline
kc3&1 &      which2&  87.3&\boxplot{  73.3}{  82.4}{  87.3}{8.1}{  95.4}\\
&2 &     micro20&  76.3&\boxplot{  56.0}{  69.8}{  76.3}{11.6}{  94.6}\\
&3 &       Bayes&  64.2&\boxplot{  45.5}{  59.2}{  64.2}{10.4}{  75.4}\\
&3  &    manualUp&  64.2&\boxplot{  50.7}{  57.5}{  64.2}{10.6}{  77.4}\\
&4 &      which4&  47.8&\boxplot{   0.0}{  40.5}{  47.8}{18.1}{  67.2}\\
&4  &  manualDown&  47.6&\boxplot{  31.3}{  39.5}{  47.6}{16.1}{  66.8}\\
&4  &      which8&  46.7&\boxplot{   0.0}{  36.2}{  46.7}{16.5}{  62.1}\\
&5 &         j48&  23.1&\boxplot{   0.0}{  13.6}{  23.1}{15.3}{  42.6}\\
&5 &        jRip&  17.7&\boxplot{   0.0}{  13.1}{  17.7}{10.8}{  54.2}\\\hline
mw1&1 &      which2&  62.4&\boxplot{  35.8}{  57.4}{  62.4}{13.4}{  83.3}\\
&1  &  manualDown&  60.2&\boxplot{  42.8}{  52.1}{  60.2}{11.6}{  71.8}\\
&1  &     micro20&  55.7&\boxplot{   0.0}{  44.1}{  55.7}{19.9}{  78.5}\\
&2 &    manualUp&  47.8&\boxplot{  37.1}{  44.0}{  47.8}{7.9}{  62.5}\\
&3 &      which4&  42.7&\boxplot{   0.0}{  25.8}{  42.7}{24}{  60.6}\\
&3 &       Bayes&  41.7&\boxplot{  19.5}{  33.1}{  41.7}{14.6}{  62.1}\\
&4 &      which8&  39.3&\boxplot{   0.1}{  35.6}{  39.3}{12}{  60.4}\\
&5 &         j48&  20.0&\boxplot{   0.0}{  10.0}{  20.0}{14.3}{  42.9}\\
&5  &        jRip&  15.8&\boxplot{   0.0}{   7.9}{  15.8}{23.3}{  49.4}\\\hline
pc1&1 &      which2&  65.0&\boxplot{   0.0}{   0.0}{  65.0}{71.1}{  81.8}\\
&1  &     micro20&  64.4&\boxplot{   0.0}{  51.0}{  64.4}{17.9}{  78.6}\\
&1  &    manualUp&  60.6&\boxplot{  52.1}{  58.4}{  60.6}{5}{  71.6}\\
&2 &       Bayes&  51.5&\boxplot{  36.4}{  46.1}{  51.5}{7.3}{  60.9}\\
&3  &  manualDown&  44.6&\boxplot{  32.3}{  41.9}{  44.6}{4.3}{  55.3}\\
&4 &      which8&  22.6&\boxplot{   0.0}{   9.1}{  22.6}{21.6}{  47.7}\\
&4  &         j48&  19.2&\boxplot{   3.1}{  12.5}{  19.2}{12.1}{  41.5}\\
&4  &        jRip&  15.1&\boxplot{   0.0}{  11.0}{  15.1}{12.2}{  30.8}\\
&4  &      which4&   0.0&\boxplot{   0.0}{   0.0}{   0.0}{50.3}{  59.0}\\\hline
&& & & \multicolumn{1}{c}{50\%} 
\end{tabular}}
\end{center}
\caption{Seven examples of pattern \#1: WHICH2 ranked \#1 {\em and} has highest median.
This figure is reported in the same format as \fig{bands}.
}\label{fig:bands1}
\end{figure}

%XXXX why 10*3
% XXX from  the pseective of different proponents

\begin{figure}[!t]
\begin{center}
{\scriptsize\begin{tabular}{l@{~}l@{~}l@{~}r@{~}c} 
\multicolumn{4}{c}{~}& \multicolumn{1}{c}{2nd quartile, median }\\
data &rank & treatment & median $C'_2$ & \multicolumn{1}{c}{and 3rd quartile} \\\hline
ar3 &1 &  manualDown&  54.7&\boxplot{   0.0}{  40.0}{  54.7}{39.3}{ 100.5}\\
&1  &    manualUp&  52.6&\boxplot{   0.0}{  26.0}{  52.6}{34.3}{  99.3}\\
& 1 &         j48&  47.8&\boxplot{   0.0}{   0.1}{  47.8}{59.1}{ 100.0}\\
&1 &      which8&  42.7&\boxplot{   0.0}{   0.1}{  42.7}{61.6}{ 100.0}\\
&1 &      which2&  40.4&\boxplot{   0.0}{  20.6}{  40.4}{31.1}{ 100.2}\\
&2 &      which4&  41.8&\boxplot{   0.0}{  24.4}{  41.8}{28}{  89.2}\\
& 2 &     micro20&  38.0&\boxplot{   0.0}{   0.4}{  38.0}{54.1}{  77.2}\\
&3 &       Bayes&  34.4&\boxplot{   0.0}{   0.4}{  34.4}{42.3}{  60.8}\\
&4 &        jRip&   0.2&\boxplot{   0.0}{   0.0}{   0.2}{35.5}{  64.1}\\\hline
ar5&1 &     micro20&  77.6&\boxplot{  31.6}{  49.9}{  77.6}{37.2}{ 100.0}\\
& 1 &      which4&  71.4&\boxplot{   0.2}{  51.1}{  71.4}{32.1}{ 100.0}\\
& 1 &  manualDown&  69.6&\boxplot{  41.3}{  51.0}{  69.6}{30.5}{ 100.5}\\
& 1 &      which2&  67.6&\boxplot{   0.0}{  44.2}{  67.6}{36.2}{ 100.0}\\
& 1 &         j48&  56.1&\boxplot{   0.1}{  35.8}{  56.1}{34.7}{ 100.1}\\
& 1 &        jRip&  55.0&\boxplot{   0.0}{   0.2}{  55.0}{73.5}{ 100.2}\\
& 1 &       Bayes&  54.1&\boxplot{  24.3}{  42.7}{  54.1}{34.8}{ 100.2}\\
&2 &    manualUp&  56.5&\boxplot{   2.0}{  24.1}{  56.5}{39.5}{  80.9}\\
&3 &      which8&   0.0&\boxplot{   0.0}{   0.0}{   0.0}{31.1}{ 100.0}\\
\hline
&& & & \multicolumn{1}{c}{50\%} 
\end{tabular}}
\end{center}
\caption{Two examples of pattern \#2: While WHICH2 did not achieve highest medians,
it was still ranked \#1 compared to eight other methods.
This figure is reported in the same format as \fig{bands}.
}\label{fig:bands2}
\end{figure}

\begin{figure}[!t]
\begin{center}
{\scriptsize\begin{tabular}{l@{~}l@{~}l@{~}r@{~}c} 
\multicolumn{4}{c}{~}& \multicolumn{1}{c}{2nd quartile, median }\\
data &rank & treatment & median $C'_2$ & \multicolumn{1}{c}{and 3rd quartile} \\\hline
mc2&1 &    manualUp&  74.3&\boxplot{  63.3}{  70.9}{  74.3}{7.4}{  80.4}\\
&2 &     micro20&  57.1&\boxplot{   0.0}{   6.3}{  57.1}{61.5}{  81.7}\\
&2  &       Bayes&  55.9&\boxplot{  21.4}{  46.6}{  55.9}{12.5}{  79.1}\\
&3 &         j48&  43.7&\boxplot{  21.9}{  29.3}{  43.7}{26.1}{  69.7}\\
&3  &  manualDown&  42.8&\boxplot{  29.7}{  38.1}{  42.8}{9.1}{  57.4}\\
&4 &        jRip&  28.5&\boxplot{  12.7}{  17.0}{  28.5}{18.2}{  56.4}\\
&5 &      which8&  21.9&\boxplot{   0.0}{  11.2}{  21.9}{16.2}{  42.4}\\
&6 &      which4&   5.6&\boxplot{   0.0}{   0.0}{   5.6}{25.3}{  47.9}\\
&6  &      which2&   0.0&\boxplot{   0.0}{   0.0}{   0.0}{40.8}{  99.7}\\
\hline
&& & & \multicolumn{1}{c}{50\%} 
\end{tabular}}
\end{center}
\caption{The only example of pattern \#3:  WHICH2 losses (badly) but  MICRO20 still ranks high.
This figure is reported in the same format as \fig{bands}.}\label{fig:bands3}
\end{figure}


