\section{Visualization of Brittleness}
This work is motivated by a recent National Academy of Sciences report titled ``Strengthening Forensic Science" \cite{09NAS}. This report took special notice of forensic interpretation models stating:

\begin{quotation}
With the exception of nuclear DNA analysis, however, no forensic method has been rigorously shown to have the capacity to consistently, and with a high degree of certainty, demonstrate a connection between evidence and a specific individual or source. \cite{09NAS}, p6
\end{quotation}

In this study we visualize the inconsistencies of four(4) of these forensic methods in one way. By simply plotting the measurements derived from evidence from a crime scene denoted as $x$ and suspect ($y$), against the results of interpretation. 

The rest of this section gives details of the four(4) forensic models evaluated in this work, followed by the visualization of these models to highlight their brittleness which results in the inconsistencies in model results.

\subsection{Glass Forensic Models}
This section provides an overview of the following glass forensic models used in this work to show brittleness.

\begin{enumerate}
\item The 1978 Seheult model \cite{Seheult78}
\item The 1980 Grove model \cite{Grove80}
\item The 1995 Evett model \cite{Evett94}
\item The 1996 Walsh model \cite{Walsh94}
\end{enumerate}

\subsubsection{Seheult 1978}
\label{subsubsection:seh}
Seheult \cite{Seheult78}, examines and simplifies Lindley's \cite{77Lindley} 6th equation for real-world application of Refractive Index (RI) Analysis. According to Seheult: 
\begin{quote}A measurement $x$, with normal error having known standard deviation $\sigma$, is made on the unknown refractive index $\Theta_1$ of the glass at the scene of the crime. Another measurement $y$, made on the glass found on the suspect, is also assumed to be normal but with mean $\Theta_2$ and the same standard deviation as $x$. The refractive indices $\Theta$ are assumed to be normally distributed with known mean $\mu$ and known standard deviation $\tau$. If $I$ is the event that the two pieces of glass come from the same source($\Theta_1$ = $\Theta_2$) and $\bar{I}$ the contrary event, Lindley suggests that the odds on identity should be multiplied by the factor
\begin{equation}
\frac{p(x,y|I)}{p(x,y|\bar{I})} \label{eq:lin1}
\end{equation}
In this special case, it follows from Lindley's 6th equation that the factor is
\begin{equation}
\frac{1+\lambda^2}{\lambda(2+\lambda^2)^{1/2}}^{-\frac{1}{2(1+\lambda^2)}\cdot(u^2-v^2)} \label{eq:lin2}
\end{equation}
Where
\begin{equation*}
\lambda = \frac{\sigma}{\tau}, \newline
u = \frac{x-y}{\sigma\sqrt{2}} ,  \newline
v = \frac{z-\mu}
{\tau(1+\frac{1}{2}\lambda^2)^{\frac{1}{2}}} , \newline
z = \frac{1}{2}(x+y) 
\end{equation*}
\end{quote}

\subsubsection{Grove 1980}
\label{subsubsection:gro}

By adopting a model used by Lindley and Seheult, Grove proposed a non-Bayesian approach based on likelihood ratios to solve the forensic problem. The problem of deciding whether the fragments have come from common source is distinguished from the problem of deciding the guilt or innocence of the suspect. To explain his method, Grove first reviewed Lindley's method. He argued that we should, where possible, avoid parametric assumptions about the underlying distributions. Hence, in discussing the respective roles of $\theta_1$ and $\theta_2$ Grove did not attribute any probability distribution to an unknown parameter without special justification. So when considering ($\theta_1$ != $\theta_2$), $\bar{I}$ can be interpreted as saying that the fragments are present by chance entailing a random choice of value for $\theta_2$. The simplified likelihood ratio obtained from the Grove's derivation is:
\begin{equation}
\frac{\tau}{\sigma}\cdot e^{\big\{\frac{-(X-Y)^2}{4\sigma^{2}} + \frac{(Y-\mu^2)}{2\tau^2}\big\}} \label{eq:gro2} 
\end{equation}

We are of course only concerned with the evidence about \emph{I} and $\bar{I}$ so far as it has the bearing on the guilt or innocence of the suspect. Grove also considered the Event of Guilty factor \b{G} in the calculation of likelihood ratio (LR). Therefore the LR now becomes 
\begin{equation} p(X,Y|G)/p(X,Y|\bar{G})\end{equation}
Here in the expansion event \b{T}, that fragments were transferred from the broken window to the suspect and persisted until discovery and event \b{A},that the suspect came into contact with glass from other source. Here p(A/{G})=p(A/$\bar{G}$)=Pa and p(T/G)= Pt. The resulting expression is %\newline

  \begin{equation} 
    \frac{P(X,Y,S|G)}{P(X,Y,S|\bar{G})} = 1+Pt\Big\{(\frac{1}{Pa}-1)\frac {p(X,Y|I)}{p(X,Y|\bar{I})}-1\Big\}\label{eq:gro4}  
  \end{equation} 
  
\subsubsection{Evett 1995}
\label{subsubsection:eve}

Evett et al used data from forensic surveys to create a Bayesian approach in determining the statistical significance of finding glass fragments and groups of glass fragments on individuals associated with a crime \cite{Evett94}.

Evett proposes that likelihood ratios are well suited for explaining the existence of glass fragments on a person suspected of a crime. A likelihood ratio is defined in the context of this paper as the ratio of the probability that the suspected person is guilty given the existing evidence to the probability that the suspected person is innocent given the existing evidence. The given evidence, as it applies to Evett's approach, includes the number of different sets of glass and the number of fragments in each unique group of glass.

The Lambert, Satterthwaite and Harrison (LSH) survey used empirical evidence to supply probabilities relevant to Evett's proposal. The LSH survey inspected individuals and collected glass fragments from each of them. These fragments were placed into groups based on their refractive index (RI) and other distinguishing physical properties. The number of fragments and the number of sets of fragments were recorded, and the discrete probabilities were published.  In particular, there are two unique probabilities that are of great interest in calculating Evett's proposed likelihood ratio.
\begin{itemize}
\item S, the probability of finding N glass \emph{fragments} per group
\item P, the probability of finding M \emph{groups} on an individual.
\end{itemize}

%\clearpage
%\subsection{Mathematical Formulae}
The following symbols are used by Evett to express his equations:
\begin{itemize}
\item $P_n$ is the probability of finding $n$ groups of glass on the surface of a person's
clothes
\item $T_n$ is the probability that $n$ fragments of glass would be transferred, retained
and found on the suspect's clothing if he had smashed the scene window
\item $S_n$ is the probability that a group of glass fragments on a person's clothing
consists of $n$ fragments
\item $f$ is the probability that a group of fragments on person's
clothing would match the control sample
\item $\lambda$ is the expected number of glass fragments remaining at a time, $t$
\end{itemize}
Evett utilizes the following equations to determine the likelihood ratio for the first case described in his 1994 paper. In this case, a single window is broken, and a single group of glass fragments is expected to be recovered.

\begin{equation}
LR = \frac{{P_0}{T_n}}{{P_1}{S_n}{f}}+{T_0}
\end{equation}

\begin{equation} 
T_n = \frac{e^{-\lambda}{\lambda^n}}{n!}
\end{equation}

\subsubsection{Walsh 1996}
\label{subsubsection:wal}

The equation presented by Walsh \cite{Walsh94} is similar to one of Evett's. The difference is that Walsh argues that instead of incorporating grouping	and matching, only grouping should be included. Walsh says this is because match/non-match is really just an arbitrary line. He examines the use of a technique in interpreting glass evidence of a specific case. This technique is as follows:
\begin{equation}
\frac{T_L P_0 p(\bar{X},\bar{Y}|S_y,S_x)}{P_1S_Lf_1}
\end{equation}
Where 

\begin{itemize}
\item
$T_L$ = the probability of 3 or more glass fragments being transferred from the crime scene to the person.
\item
$P_0$ = the probability of a person having no glass on their clothing
\item
$P_1$ = the probability of a person having one group of glass on their clothing
\item
$S_L$ = the probability that a group of glass on clothing is 3 or more fragments
\item
$\bar{X}$ and $\bar{Y}$ are the mean of the control and recovered groups respectively
\item
$S_x$ and $S_y$ are the sample standard deviations of the control and recovered groups respectively
\item
$f_1$ is the value of the probability density for glass at the mean of the recovered sample
\item
$p(\bar{X},\bar{Y}|S_y,S_x)$ is the value of the probability density for the difference between the sample means
\end{itemize} 
%%%%

\subsection{Visualization of Brittleness in Models}
%\subsubsection{The First Technique}

The result of applying the visualization technique i.e. plotting the measurements derived from evidence from a crime scene denoted as $x$ and suspect ($y$), against the results of interpretation on the glass forensic models are shown in \fig{models}.

\begin{figure}[h!]
  \begin{center}
  \scalebox{0.97}{
    \begin{tabular}{c}
      \resizebox{60mm}{!}{\includegraphics{seheult-chart}} \\
      \resizebox{60mm}{!}{\includegraphics{grove-chart}} \\
      \resizebox{60mm}{!}{\includegraphics{walsh-chart}} \\
      \resizebox{60mm}{!}{\includegraphics{evett-chart}} \\
    \end{tabular}}
    \caption{Visualization of four(4) glass forensic models}
    \label{fig:models}
  \end{center}
\end{figure}

For the first two(2) models the $x$ and $y$ axes represent the mean refractive index (RI) values of evidence from a crime scene and suspect respectively. While the $x$ axis of the Walsh model represents $f1$ is the value of the probability density for glass at the mean of the recovered sample and the $y$ axis represents the value of the probability density for the difference between the sample means. The $x$ and $y$ axes of the Evett model represents $\lambda$ and $f-values$ respectively. The $z$ axis of all the models represent the likelihood ratio (LR) generated from these models, in other words, the significance of the match/non-match of evidence to an individual or source.

Using data donated by the Royal Canadian Mounted Police (RCMP), values such as the RI ranges and their mean, were extracted to generate random samples for the forensic glass models. In all four(4) models $1000$ samples are randomly generated for the variables in each model. For instance, in the Seheult model, each sample looks like this: [$x$, $y$, $\sigma$, $\mu$, $\tau$]. The symbols are explained in \ref{subsubsection:seh}.

In \fig{models} - the sehult and grove models, brittleness or Smalldon's ``fall-off-the-cliff-effect" is clearly demonstrated. These models proposed by Seheult (Section \ref{subsubsection:seh}) and Grove (\ref{subsubsection:gro}) respectively, show how the likelihood ratio changes (on the vertical axis) as we try different values from the refractive index of from glass from two sources (x and y). This model could lead to incorrect interpretations if minor errors are made when measuring the refractive index of glass samples taken from a suspect's clothes. Note how, near the region where x=y, how tiny changes in the x or y refractive indexes can lead to dramatic changes in the likelihood ratio (from zero to one).

The visualization of the Evett (\ref{subsubsection:eve}) and Walsh (\ref{subsubsection:wal}) models show similar brittleness when the likelihood ratios are 0 and 1. For Walsh, values located at the edge of a cliff a LR=1 can easily become LR=0 at the smallest change in the $f1$ or $p$ values. While Evett will cause problems because a small change occurs with any sample it is possible for the LR to change. 

From these visualizations it is obvious that the concern of the National Academy of Sciences report \cite{09NAS} mentioned earlier in this section is a valid one. So how can this concern be alleviated? We propose not only including a \emph{brittleness} measure to a forensic method as a solution, but also moving away from forensic models which use a Bayesian approach \cite{Seheult78, Evett84, Evett90, Evett94, Walsh94}, and statitical assumptions \cite{Seheult78, Grove80, Walsh94}.

The following sections gives details of the models used in this work as well as the data set used to evaluate the models.