\frame { \frametitle{Sound Bites}
\begin{columns}[t]
\begin{column}[T]{1in}
\includegraphics[width=1in]{well.pdf}
\end{column}
\begin{column}[T]{3.5in}
\bi
\item 
Static code features can be used to build software quality predictors.
\item But they are a \underline{shallow well}.
	\bi
 	\item We can easily get to the bottom.
	\item But further effort will not take us deeper.
	\ei
\item
Unless we change the \underline{rules} of the game.
\ei
\end{column}
\end{columns}
}
\frame { \frametitle{Acknowledgments}
\begin{columns}
\begin{column}[T]{1in}
\begin{center}
\tiny\includegraphics[width=0.5in]{nsf.png}

National Science Foundation, USA


\includegraphics[width=0.5in]{tubitak.png}

Scientific and Technological Research Council of Turkey
\end{center}
\end{column}
\begin{column}[T]{3.5in}
\scriptsize
\bi
\item WVU
	\bi\scriptsize
	\item Bojan Cukic,  Zach Milton, Gregory Gay, \newline Yue Jiang, Justin DiStefano,
	\item Supported by NSF grant \#CCF-0810879 
\ei
\item Softlab (Bo{\u g}azi{\c c}i  University, Turkey)
	\bi\scriptsize
	\item Ay\c{s}e Bener, Burak Turhan 
	\item Supported by Turkish Scientific Research Council\newline (Tubitak EEEAG \#108E014).
\ei
\item PSU: Jeremy Greenwald 
	

\ei
\end{column}
\end{columns}
\scriptsize
~\\
~\hrule~

Reference herein to any specific commercial product, process, or service 
by trademark, manufacturer, or otherwise, does not constitute or imply its 
endorsement by the Governments of Turkey or the United States.
}
\frame {\frametitle{PROMISE '09}
\begin{columns}
\begin{column}[T]{2in}
\includegraphics[width=2in]{promise.png}
\end{column}
\begin{column}[T]{2.5in}
\bi\scriptsize
\item \url{www.promisedata.org/2010}
\item Motto:
\bi \scriptsize
\item Repeatable, refutable, improvable
\item Put up or shut up
\ei
\item Papers ...
\bi\scriptsize
\item ... and the data used to generate those papers
\item \url{www.promisedata.org/data}
\ei
\ei
\end{column}
\end{columns}
}
	\section{What is ``Defect Prediction''?}
\frame { \frametitle{Problem}
\begin{columns}\begin{column}[T]{1.6in}
\includegraphics[width=1.5in]{blindSpots.png}
\end{column}\begin{column}[T]{3in}
\scriptsize
100\% quality assurance (QA) is infinitely expensive
\bi
\item \scriptsize
$Tests(Conf,ErrRate)=\frac{log(1 - Conf)}{log(1 - ErrRate)}$ 
\ei
Test engineers, skew QA towards what is most critical:
\bi\scriptsize
\item e.g. model checking restricted to the kernel of the guidance system
\ei

But if we only look at \{A,B,C\}....
\bi
\item ... what hides in the blind spot of \{E,F,G,...\}?
\ei
Proposal: 
\bi\item\scriptsize
Augment focused expensive QA methods with cheaper faster methods
\item
But how to sampling methods blind spots?
\bi\scriptsize
\item quickly
\item cheaply
\ei
\ei
\end{column}
\end{columns}
}
\frame { \frametitle{One Solution: Data Mining}
\bi\scriptsize
\item Input: rows of data
\begin{center}\scriptsize
\begin{tabular}{ccc|c}
\multicolumn{3}{c}{features} & class\\\hline
name & age & shoeSize & mood\\\hline
tim & 48 & 11 & happy\\
tim & 12 & 5 & sad\\
tim & 28 & 9 & happy \\
tim & 100 & 11 & happy
\end{tabular}
\end{center}
\item Learners: 
\bi\scriptsize
\item Naive Bayes: statistical feature combinations for class prediction
\item RIPPER : learns rules
\item C4.5 : decision tree learner
\item Random Forests : learn 100s of trees
\item etc
\ei
\item Output: combinations of  features that predict for the class
\ei
}
\frame { \frametitle{Solution: the Details}
\bi
\item Using data mining: explore the ``modules'' 
\bi
\item smallest functional unit $\in \{function, method, class, ...\}$
\ei
\item Take logs of modules described as static code modules:
	\bi
	\item Lines of code and comment measures
	\item Counts of intra-module symbols (\cite{halstead77});
	\item Measures of intra-module call graphs (\cite{mccabe76});
	\ei
\item Join the logs to number of defects seen in each module 
	\bi
	\item often, discretized to $\{yes,no\}$;
	\ei
\item Find feature combinations that predict for $defective\in \{yes,no\}$;
\ei
}
\frame {\frametitle{Examples of Static Code Features}

\begin{center}{\tiny
\begin{tabular}{l@{~}|l@{~}|l@{~}|c@{~}p{3.9cm}|}\cline{2-5}
		&\multicolumn{2}{l}{m = Mccabe }       &$v(g)$  &cyclomatic\_complexity  \\
        &        \multicolumn{2}{l}{~}                                 & $iv(G)$&design\_complexity\\
        &         \multicolumn{2}{l}{~}                                & $ev(G)$&essential\_complexity\\\cline{2-5}
        &locs&{loc}                         &   &loc\_total (one line = one count\\\cline{3-5}
	&	&loc(other)                      &   &loc\_blank           \\
        &       &                                &   &loc\_code\_and\_comment\\
        &       &                                &   &loc\_comments         \\
        &        &                                &   &loc\_executable\\
        &       &                                &    &number\_of\_lines (opening to closing brackets)\\\cline{2-5}
	&{Halstead} & h& $N_1$ &num\_operators\\
        &       &                                 &$N_2$ &num\_operands \\
        &       &                                 &$\mu_1$ &num\_unique\_operators\\
        &       &                                 &$\mu_2$ &num\_unique\_operands\\\cline{3-5}
        &       &  { H}&$N$  &length: $N= N_1 + N_2$\\
        &       &                                 & $V$  &volume: $V=N * log_{2}\mu$\\
        &       &                                 & $L$  &level: $L=V^*/V$ where\newline $V^*=(2+\mu{_2}^{*})log_{2}(2+\mu{_2}^{*})$\\
        &       &                                 & $D$  &difficulty: $D=1/L$\\
        &       &                                 &$I$   &content: $I=\hat{L}*V$ where\newline $\hat{L}=\frac{2}{\mu_{1}} * \frac{\mu_{2}}{N_{2}}$ \\
        &       &                                 & $E$  &effort: $E=V/\hat{L}$\\
        &       &                                 & $B$  &error\_est\\
        &       &                                 & $T$  &prog\_time: $T=E/18$ seconds\\\cline{2-5}
\end{tabular}}
\end{center}
}
\frame { \frametitle{Why Do That?}
\footnotesize
\begin{columns}[t]
\begin{column}[T]{5.5cm}
{\em Useful:} out-performs known industrial baselines at defect detection:
\bi
\item IEEE Metrics'02 panel: manual software reviews finds ${\approx}60\%$ (\cite{shu02});
\item Raffo (pers. comm.): industrial review methods find  \mbox{$pd=TR(35, 50, 65)\%$} 
\item Data mining static code features finds (median) 71\%\newline (\cite{me07b}).
\ei
\end{column}
\begin{column}[T]{5.5cm}
{\em Easy to use}:
\bi
\item Automatic, cheap and fast to collect, scales to large systems.
\item Manual methods: 8-20 LOC/minute for manual code reviews.
\ei

{\em Widely used}:
\bi
\item Hundreds of research papers. Earliest:~\cite{porter90}?
\item Recommended by numerous texts
\item  Large government contractors: only review modules that trigger
a static code analyzer
\ei
\end{column}
\end{columns}
}
\frame { \frametitle{Why Use Static Code Features?}
\bi
\item Why not use...
	\bi
	\item knowledge of the developers (\cite{nach08})
	\item  Or history of runtime defectives (\cite{musa87})
	\item Or XYZ?
	\ei
\item A: Use whatever is available
	\bi
	\item And that changes from site to site
\ei
\item I dream of the day that I work with an organization with stable products and practices.
\bi\item Meanwhile, in the real world...
\ei
\ei
}
\subsection{\cite{me08a}}
\frame { \frametitle{Dealing with Organizational Change }
{\tiny
\begin{tabular}{r|r@{~}l|p{1.5in}|l}
year & & \#IV\&V project  &   notes & oversight\\\hline
1988 &n/a & &Space shuttle begins IV\&V&Johnson (Texas)\\\hline
1991 &n/a& &New the IV\&V facility & Headquarters (east  coast)\\
1994 &1  &\rule{0.5mm}{1.5mm} &International space station IV\&V begins &\\
1995& 1 &\rule{0.5mm}{1.5mm}&&\\\hline
1996& 2 &\rule{1mm}{1.5mm}&&NASA Ames (west coast)\\
1996& 3 &\rule{1.5mm}{1.5mm}&
        IV\&V funded from project budgets &\\
1997& 3 &\rule{1.5mm}{1.5mm}&&\\
1998& 3 &\rule{1.5mm}{1.5mm}&&\\
1999& 12 &\rule{6mm}{1.5mm}& IV\&V now considered on all software&\\\hline
2000& 15&\rule{7.5mm}{1.5mm}& & GSFC (east coast)\\
2001& 20 &\rule{10mm}{1.5mm}&&\\
2002& 36&\rule{18mm}{1.5mm}&&\\
2003& 42 &\rule{21mm}{1.5mm}&IV\&V funded from  central agency source. &\\
2004& 37&\rule{13mm}{1.5mm}&&\\
2005& 24&\rule{12mm}{1.5mm}&SILAP data collection begins&\\
2006& 26&\rule{13mm}{1.5mm}&&\\
2007& 24&\rule{12mm}{1.5mm}&SILAP data collection ends&\\
\end{tabular}
}
\begin{itemize}
\item 2003: Loss of Columbia $\Rightarrow$ ``return to flight'' reorganization
\item 2004: Bush's new vision for space exploration
\item Always:  layers of contractors;  so ``oversight'', not ``insight''
\end{itemize}
}
\subsection{}
\frame { \frametitle {The Real Question}
\bi
\item
Not what features ``are right'';
\bi
\item
				But what features are available ``right now''.
\ei
\item Particularly when you can not control data collection
\bi
\item Agile 
\item Out-source
\item Open source
\item Sub-contractors
\item Your current project?
\ei
\item
Sometimes, all you can access ``right now'' is source code.
\ei
}
\section{Value  of Defect Prediction}
\subsection{\cite{fenton97,shepperd94}}
\frame { \frametitle{Aren't Static Code Features.... Stupid?}
\begin{columns}
\begin{column}[T]{2.8in}

\begin{center}{\tiny
\begin{tabular}{l@{~}|l@{~}|l@{~}|c@{~}p{3.9cm}|}\cline{2-5}
		&\multicolumn{2}{l}{m = Mccabe }       &$v(g)$  &cyclomatic\_complexity  \\
        &        \multicolumn{2}{l}{~}                                 & $iv(G)$&design\_complexity\\
        &         \multicolumn{2}{l}{~}                                & $ev(G)$&essential\_complexity\\\cline{2-5}
        &locs&{loc}                         &   &loc\_total (one line = one count\\\cline{3-5}
	&	&loc(other)                      &   &loc\_blank           \\
        &       &                                &   &loc\_code\_and\_comment\\
        &       &                                &   &loc\_comments         \\
        &        &                                &   &loc\_executable\\
        &       &                                &    &number\_of\_lines (opening to closing brackets)\\\cline{2-5}
	&{Halstead} & h& $N_1$ &num\_operators\\
        &       &                                 &$N_2$ &num\_operands \\
        &       &                                 &$\mu_1$ &num\_unique\_operators\\
        &       &                                 &$\mu_2$ &num\_unique\_operands\\\cline{3-5}
        &       &  { H}&$N$  &length: $N= N_1 + N_2$\\
        &       &                                 & $V$  &volume: $V=N * log_{2}\mu$\\
        &       &                                 & $L$  &level: $L=V^*/V$ where\newline $V^*=(2+\mu{_2}^{*})log_{2}(2+\mu{_2}^{*})$\\
        &       &                                 & $D$  &difficulty: $D=1/L$\\
        &       &                                 &$I$   &content: $I=\hat{L}*V$ where\newline $\hat{L}=\frac{2}{\mu_{1}} * \frac{\mu_{2}}{N_{2}}$ \\
        &       &                                 & $E$  &effort: $E=V/\hat{L}$\\
        &       &                                 & $B$  &error\_est\\
        &       &                                 & $T$  &prog\_time: $T=E/18$ seconds\\\cline{2-5}
\end{tabular}}
\end{center}


\end{column}
\begin{column}[T]{1.45in}
\bi\scriptsize
	\item  $v(g)$ correlated to LOC 
	\item ``For a large class of software ($(v(g)$) is no more than a proxy
	for, and in many cases outperformed by, lines of code'' \newline
-- Shepperd \& Ince
\ei
\end{column}\end{columns}
}
\subsection{\cite{turhan08}}
\frame { \frametitle{ So What Would Happen if... }
\scriptsize
We learned defect predictors from NASA aerospace applications
\bi
\item 
Then applied them to software developed for Turkish whitegoods?
\item
(Caveat: in both studies, same data dictionary but different data.)
\ei
{\tiny \begin{center}
\begin{tabular}{l|lll}
\scriptsize
&     &       &   \\
source
&project & language & description \\ \hline
NASA&cm1  & C++ &Spacecraft instrument\\
NASA&pc1  & C++ &Flight software for earth orbiting satellite\\
NASA&mc2  & C++ &Video guidance system \\
NASA&mw1  & C++ &A zero gravity experiment related to combustion\\\hline
NASA&kc1  & C++ &Storage management for ground data\\
NASA&kc2  & C++ &Storage management for ground data\\
NASA&kc3  & JAVA &Storage management for ground data\\
\hline
SOFTLAB & ar5 & C &Washing machine \\
SOFTLAB& ar3& C &Dishwasher \\
SOFTLAB& ar4 & C &Refrigerator \\
\end{tabular}
\end{center}}
Begin digression: how do we measure performance?

}
\frame { \frametitle{Performance Measures }
\begin{columns}\begin{column}[T]{2in}
\scriptsize
$\{A,B,C,D\}$ =
true negatives, 
false negatives, 
false positives, and 
true positives
(respectively) found by a binary detector. 


\[
\begin{array}{ll}
pd=recall=&\frac{D}{B+D}\\
pf=&\frac{C}{A+C}\\
prec=precision=&\frac{D}{D+C}\\
acc=accuracy=&\frac{A+D}{A+B+C+D}\\
neg/pos=&\frac{A+C}{B+D}
\end{array}
\]
\end{column}
\begin{column}[T]{2.5in}
\scriptsize
For large $neg/pos$ values:
can be accurate and still miss most things

{\scriptsize 
\begin{center}
\begin{tabular}{r|r|r|r|}
\multicolumn{2}{c}{}&\multicolumn{2}{c }{module found in defect
logs?}\\\cline{3-4} \multicolumn{2}{c|}{}&no&yes\\\cline{2-4}
 signal&no &A = 395
 &B = 67\\\cline{2-4}
detected?&yes &C = 19
&D = 39\\\cline{2-4}
\end{tabular}
\begin{eqnarray}\nonumber
Acc=&accuracy=&83\%\\\nonumber
pf=&Prob.falseAlarm=&5\%\\\nonumber
pd=&Prop.detected=&37\%\\\nonumber
\end{eqnarray}
\end{center}
\begin{raggedleft}

End digression.

\end{raggedleft}
}

\end{column}
\end{columns}
}
\subsection{\cite{turhan08}}
\frame { \frametitle{ Results}
\scriptsize
For learner=Naive Bayes (why? see later), try ``round robin'' or ``self-'' learning
\bi\scriptsize
\item RR= ``Round robin'' : train on {\em them}, test on {\em me}.
\item `Self'' : train and test on {\em me}
\ei
{\tiny
\begin{tabular}{r|r|r|r|l|l|}
   \multicolumn{4}{c}{~} & \multicolumn{2}{c}{Probability}\\\cline{5-6}
&                         &               &       &  & False\\
&          Data source               &          Train            & Test        & Detection & Alarm \\\hline
RR&Imported data            &   all $X$ - $X_i$         & $X_i$       &  94  ( ! )       & \textcolor{red}{68}\\
self&Local data               &$90\%\;of\;X_i$                  & $10\%\;of\;X_i$   &  75        & 29\\      
RR& Filtered imported data &    k-nearest of (all $X$ - $X_i$) & $X_i$       &  69        & 27\\
\hline
\end{tabular}}
\bi
\item Best data source: local data
\item Adequate: using imported data (filtered with nearest neighbor)
\ei
Recommendations:
\bi\scriptsize
\item If no data, start  local collection 
\item Meanwhile, use imported data, filtered with nearest neighbor
\ei

Question: how much data is needed to build local detectors?

}
\section{Variance and Ceiling Effects}
\subsection{\cite{me07b}}
\frame { \frametitle{What is ``The'' Best Feature?}
\scriptsize
Experiments with N\aive Bayes: 
\bi\scriptsize
\item
While performance improved, add features. 
\ei
~\\

\begin{columns}
\begin{column}{3.5cm}
\tiny
\begin{tabular}{l|rr|c}
&  & &   index of\\
&  & &  selected \\
data& pd & pf &  feature \\\hline
pc1   & 48 & 17 &  3, 35, 37\\
mw1   & 52 & 15 &  23, 31, 35\\
kc3   & 69 & 28 & 16, 24, 26\\
cm1   & 71 & 27  & 5, 35, 36\\
pc2    & 72 & 14 & 5, 39\\
kc4   & 79 & 32 &  3, 13, 31\\
pc3   &80 & 35 & 1, 20, 37\\
pc4   &98  & 29 & 1, 4, 39\\\hline
all   & 71& 25 
\end{tabular}
\end{column}
\begin{column}[L]{5cm}
\tiny
\begin{tabulary}{\textwidth}{llll}
ID&used in &what& type\\\hline
1&2&  {loc\_blanks}  & locs\\
3&2 &  {call\_pairs} & misc\\
4&1 &  {loc\_code\_and\_command} & locs\\
5&2 &  {loc\_comments} & locs\\
13&1 &  {edge\_count}   & misc\\
16&1 &  {loc\_executable} & locs\\
20&1 &  {I} & H'\\
23&1 &   {B}& H'\\
24&1 &   {L}& H'\\
26&1 &   {T}& H'\\
31&2 & {node\_count} & misc\\
35&3 & {$\mu_2$}& h\\
36&1 & {$\mu_1$}& h\\
37&2 &  {number\_of\_lines} & locs\\
39&2 & {percent\_comments}& misc\\\hline
\end{tabulary}

H' = derived Halstead

h = raw Halstead
\end{column}\end{columns}
\scriptsize
~\\~\\
Why no single most valuable feature?
}
\subsection{\cite{me07b}}
\frame { \frametitle{Feature Information Variance}
\scriptsize
10 * \{ 90\% sample, compute ``info gain'' of each feature\} 
\begin{center}
\includegraphics[width=2.25in]{infogain.pdf}
\end{center}

\bi
\item High left-hand-side plateau (multiple candidate ``best'' features)
\item Low right-hand-side valley (small set of ``always worst'' features)
\ei

Never again: $v(g) > 10$.
}
\subsection{\cite{jiang07}; \cite{lessmann09}}
\frame { \frametitle{Value of Better Data Miners?}
\scriptsize
Ceiling effects:
	an inherent upper bound on the performance of our data miners
\bi
\item Have yet to improve our mid-2006 defect predictors: ~\cite{me07b}
\ei

\begin{columns}
\begin{column}{2in}
\includegraphics[height=1.5in]{jiang.png}

 6/9 methods are ``best''
\end{column}
\begin{column}{2in}
\includegraphics[height=1.5in]{all.pdf}

 14/19 methods are ``best'' 
\end{column}
\end{columns}

~\\
~\hrule~
~\\

Note: evaluation bias- area under the curve of a detection vs false alarm plot 
\bi\item\scriptsize AUC(PD, PF)\ei
}

\subsection{\cite{me08d}}
\frame { \frametitle{Value of More Data?}

\scriptsize  Learner = NB
Randomized order ; train on ``X''; test on next 100

\includegraphics[angle=90,height=.7in]{incrementalKC1.pdf}
\includegraphics[angle=90,height=.7in]{incrementalPC1.pdf}

Micro-sampling: $N$ defective + $N$ non-defective, $N \in \{25, 50, 75, ...\} $

\includegraphics[angle=90,height=.7in]{underKC1.pdf}
\includegraphics[angle=90,height=.7in]{underPC1.pdf}

~\\

Statistically, little gain after 100 (random) or 50 (micro-sampling)

}
\subsection{\cite{lessmann09}}
\frame { \frametitle{End of the Line?}
\begin{quote}
``...the
importance of the (learner) is less than generally assumed and
practitioners are free to 
choose from a broad set of candidate models when building defect predictors.''
\newline
{\em -- Lessmann et al.}

\end{quote}
~\\

No value in new algorithms for defect prediction?
\bi
\item Not unless we change the rules of the game
\ei
}

