\section{Changing the Rules }
%\frame{\tableofcontents}
\subsection{}
%\subsection{\cite{me07e}}
%\frame { \frametitle{Precision as (useful,useless) as Accuracy}
%\bi
%\item
%Accuracy misleading when $neg/pos$ is large (target class is rare)
%\item
%So is precision: 
%\item
%Recall the Zhang \& Zhang equation:
%$pf = \frac{pos}{neg}\cdot\frac{ (1 -  prec) }{ prec}\cdot recall$
%\item
%Standard values for $neg/pos$ in NASA data sets: 
%\bi
%\item
%\{1.04, 
%7.33,
%9,
%10.11,
%13.29,
%15.67, 
%249\}
%\ei
%\ei
%\begin{center}
%\includegraphics[height=1in]{y0.pdf}~~~~
%\includegraphics[height=1in]{lmeansv.pdf} 
%\end{center}
%}
%\frame { \frametitle{For $neg/pos \gg 1$: Hard to Be Precise \& Accurate}
%~~~~~\includegraphics[width=1.75in]{y0.pdf}~~~~
%\includegraphics[width=1.75in]{y05.pdf} 
%\begin{itemize}
%\item Not solved by bagging, boosting, random forests (\cite{ma07}).
%\item Only known counter-example in the SE literature:
%\begin{itemize}
%\item Hayes, Dekhtyar, Senthil, TSE'06 (\cite{hayes06})
%\item
%High precision \& recall after 4 rounds of user feedback;
%\item
%But {\em over all} the examples seen in all four rounds, 
%\bi
%\item Over all precision very low ($\approx 20\%$) (\cite{dekhtyar07}).
%\ei
%\ei
%\end{itemize}
%}
\subsection{\cite{me07e}}
\frame { \frametitle{Generalized Evaluation Bias}
\vspace{-1cm}
\begin{center}
\includegraphics[width=2in]{locplatonic.pdf}
\end{center}

{\tiny
\begin{tabular}{p{6cm}|c|c|c}
	& pd & pf & effort$^\#$\\\hline

Risk adverse (e.g. airport bomb detection, morning sickness)   & hi& hi&        \\\hline
Cost adverse (e.g. budget conscious)          & med   & lo    &\\\hline
\cite{arisholm06}  & & & $<pd$\\\hline
\end{tabular}

~\\

$^\#$effort = LOC in the modules predicted to be faulty}
}
\subsection{\cite{milton08}}
\frame { \frametitle{Evaluation-aware Learners }
All learners have an search bias $S$ and an evaluation bias $E$. e.g. C4.5:
\bi
\item $S$ = infogain
\item $E$ = pd,pf,accuracy, etc
\item Note: usually, $S \not= E$
\ei
Question: What if we make $S = E$ ?
\bi
\item Answer: \cite{milton08}
\ei
}
\subsection{\cite{milton08}}
\frame { \frametitle{Evaluation-aware Learning with ``WHICH'' }
\vspace{-0.5cm}
\begin{center}
\begin{columns}
\begin{column}[T]{2.3in}
\scriptsize
\begin{enumerate}\scriptsize
\item Discretize all numeric features. 
\item Sort all ranges using $E$ onto a stack
\item Pick any 2 items near top-of-stack
\item Combine items, score them with $E$, insert them into the sorted stack.
\item Goto 3
\end{enumerate}
\scriptsize

Note: no $S$; $E$ is customizable.



\end{column}\begin{column}[T]{2.4in}\scriptsize
\vspace{-0.5cm}\begin{center}\includegraphics[width=1.5in]{picks.pdf}

\scriptsize
Top of stack stabilizes quickly  (UCI data).
\end{center}
\end{column}\end{columns}
\end{center}
\scriptsize Other methods can bias learner predictions:
\bi
\scriptsize
\item Apply $E$ during decision tree splitting
\item \cite{elkan01}: cost-sensitive learning (*)
\item \cite{fawcett01}: ROC ensemble combinations (*)
\ei 
(*) But what work if search criteria 
is orthogonal to the evaluation criteria?

}
\frame { \frametitle{Experiments with ``WHICH''}
\begin{center}
	\vspace{-0.4cm}\begin{columns}
		\begin{column}[T]{7cm}
			\scriptsize
\cite{arisholm06} 
\bi
\item\scriptsize
For a budget-conscious team, 
\item if X\% of modules predicted to be faulty 
\item but they contain $\le X\%$ of the detects,
\item then that defect predictor is not useful
\item i.e. they prefer $pd > effort$
\ei

~\\

Operationalizing their bias:
\bi\scriptsize
\item Find modules triggered by the learner
\item Sort them in ascending order of size
\item Assume human inspectors find $\Delta$ of the defects in the triggered modules
\item Score learner as ratio of ``best'' effort-vs-pd curve
\bi\scriptsize
\item ``best'' only triggers on defective modules
\item Note : $\Delta$ cancels out
\ei
\ei
\end{column}\begin{column}[T]{4cm}
\includegraphics[width=4cm]{effort1.pdf}


\begin{raggedleft}

~\\

\scriptsize
``bad'' : worse than manual 

``good'' : beats manual 

\end{raggedleft}
\end{column}
\end{columns}\end{center}
}
\frame { \frametitle{Experiments with ``WHICH''}
\vspace{-0.5cm}
\bi
\scriptsize
\item
10 random orderings * 3-way cross-val
\item
10 sets of static code features from NASA, Turkish whitegoods 
\item
``Rank'' computed using Mann-Whitney U test (95\%)
\item 
Micro20: training on 20 defective + 20 non-defective
\ei

\begin{center}
{\tiny\begin{tabular}{l@{~}l@{~}l|@{~}r|@{~}c} 
 &rank & treatment & median ``best''\% & \multicolumn{1}{c}{2nd quartile, median, 3rd quartile} \\\hline
&1 &      WHICH&  87.3&\boxplot{  73.3}{  82.4}{  87.3}{8.1}{  95.4}\\
&2 &     micro20&  76.3&\boxplot{  56.0}{  69.8}{  76.3}{11.6}{  94.6}\\
&3 &       NB&  64.2&\boxplot{  45.5}{  59.2}{  64.2}{10.4}{  75.4}\\
&3  &    manual&  64.2&\boxplot{  50.7}{  57.5}{  64.2}{10.6}{  77.4}\\
&4 &         C4.5&  23.1&\boxplot{   0.0}{  13.6}{  23.1}{15.3}{  42.6}\\
&4 &        jRip&  17.7&\boxplot{   0.0}{  13.1}{  17.7}{10.8}{  54.2}\\\hline

%& 1 &      WHICH&  70.9&\boxplot{   0.0}{  52.5}{  70.9}{27}{ 100.2}\\
%& 1 &     micro20&  67.0&\boxplot{   0.0}{  50.0}{  67.0}{25.9}{ 100.0}\\
%&2 &    manualUp&  61.1&\boxplot{   0.0}{  52.0}{  61.1}{16.4}{  99.3}\\
%&3 &       N\aive Bayes&  56.8&\boxplot{   0.0}{  47.5}{  56.8}{15.6}{ 100.2}\\
%&4 &         C4.5&  27.6&\boxplot{   0.0}{  16.9}{  27.6}{26.3}{ 100.1}\\
%&5 &        RIPPER&  21.9&\boxplot{   0.0}{  10.6}{  21.9}{26.3}{ 100.2}\\\hline
&& & & \multicolumn{1}{c}{50\%} 
\end{tabular}}
\end{center}
\scriptsize

Shallow well: we do not need much data to do it (40 examples).
}
%% \frame { \frametitle{Why did it Work so Well?}
%% \scriptsize
%% Recall a previous slide:

%% \begin{columns}
%% \begin{column}{2in}
%% \includegraphics[height=1.5in]{jiang.png}

%%  6/9 methods are ``best''
%% \end{column}
%% \begin{column}{2in}
%% \includegraphics[height=1.5in]{all.pdf}

%%  14/19 methods are ``best'' 
%% \end{column}
%% \end{columns}

%% ~\\
%% ~\hrule~
%% ~\\

%% Most learners developed for classification, well suited to maximizing
%% AUC(Pd, Pf).
%% \bi\scriptsize
%% \item
%% So most learners fail to impress when we change the rules.
%% \ei
%% New line of research:\bi\scriptsize \item tuning  learners to local business case
%% \ei
%% }

\section{Conclusions \& Next Steps}
\subsection{}
\frame { \frametitle{But Why does it Work so Well?}
Related question: 
\bi
\item Why does IR work so well?
\item Same  answer for  both questions?
\ei
Is our technical work constrained by:
\bi
\item
compilers, \item
languages, \item
target domains, 
\item human short-term memory, \item
etc
\ei
What is this invisible hand?
}
\frame { \frametitle{F.A.Q. }
Are defect predictors  useful? 
\bi\item As a method to reduce the cost of more expensive QA,   yes.
\ei
Are defect predictors general? 
\bi \item  Yes: after NN,  NASA \& SOFTLAB's predictors work on each other's site
\ei
But learning from local data is best. \bi 
\item How much data is needed?
\bi
\item Dozens to under 100  for micro-sampling to random sampling
\ei
\ei
}
\frame { \frametitle{F.A.Q. (more)}
Which learner is best for building defect predictors?
\bi \item  When maximizing PD-vs-PF, there are many choices.
\item Otherwise, tune learner to local evaluation bias (e.g. WHICH)
\ei
What is the ``best'' feature?
\bi
\item Wrong question. ``Best'' is domain-specific.
\item Collect everything that is readily and cheaply  available
\ei
}
\frame { \frametitle{Research Directions: NSF CCF funding}
We have seen above that the ``best'' features are data set dependent.
\bi
\item Due to feature information variance
\ei
So lets look harder at exploiting local knowledge.
\bi
\item Microsoft example
\ei
Given the small samples needed for learning detectors (dozens to 100)
\bi
\item Augment (? replace) data mining
\item ... with human-in-the-loop case-based-reasoning
\ei
Do you think that with your domain expertise you can do better than stupid static features?
\bi
\item Then lets talk some.
\ei
}
\frame { \frametitle{And Finally.. }
\begin{columns}[t]
\begin{column}[T]{1in}
\includegraphics[width=1in]{well.pdf}
\end{column}
\begin{column}[T]{3.5in}
\bi
\item 
Static code features can be used to build software quality predictors.
\item But they are a \underline{shallow well}.
	\bi
 	\item  Easily get to the bottom \newline \textcolor{red}{With $\le 100$ examples}.
	\item Further effort will not take us deeper \newline
				\textcolor{red}{Ceiling effects on AUC(Pd,Pf)}.
	\ei
\item
Unless we change the \underline{rules} of the game:
\bi
\item \textcolor{red}{Using evaluation aware learning}
\item \textcolor{red}{Augment data mining with human-in-the-loop}
\ei
\ei
\end{column}
\end{columns}
}

