%&pdflatex; \documentclass{elsarticle} %\usepackage[colorlinks=true]{hyperref} \usepackage{graphicx} \usepackage{multirow} \usepackage{lscape} %\usepackage[nolists]{endfloat} \usepackage{endfloatsideways} \def\url#1{\texttt{#1}} \begin{document} \begin{frontmatter} \title{Learning Patterns of University Student Retention} \author[rvt]{Ashutosh Nandeshwar\corref{cor1}} \ead{anandesh@kent.edu} \ead[url]{www.nandeshwar.info} \author[focal]{Tim Menzies} \ead{tim@menzies.us} \ead[url]{www.menzies.us} \author[els]{Adam Nelson} \ead{rabituckman@gmail.com} \cortext[cor1]{Corresponding author} \address[rvt]{Kent State University, 126 Lowry Hall, Kent, OH 44242 Phone: 330-672-82222} \address[focal]{West Virginia University, 841a Engineering Sciences Building, Morgantown, WV 26505} \address[els]{West Virginia University, Engineering Sciences Building, Morgantown, WV 26505} \begin{abstract} Learning predictors for student retention is very difficult. After reviewing the literature, it is evident that there is considerable room for improvement in the current state of the art. As shown in this paper, improvements are possible if we (a)~explore a wide range of learning methods; (b)~take care when selecting attributes; (c)~assess the efficacy of the learned theory not just by its {\em median} performance, but also by the {\em variance} in that performance; (d)~study the {\em delta} of student factors {\em between} those who stay and those who are retained. Using these techniques, for the goal of predicting if students will remain for the first three years of an undergraduate degree, the following factors were found to be informative: family background and family's social-economic status, high school GPA and test scores. \end{abstract} \begin{keyword} data mining \sep student retention \sep predictive modeling \sep financial aid \end{keyword} \end{frontmatter} %\input{intro} \section{Introduction} This article uses data mining to find patterns of student retention at American Universities. Such an analysis is urgently required. In our work, we have seen a disconnect between accepted best practices and the data available to support those practices: \begin{itemize} \item Based on our discussion with the university administrators, we assert that there is much {\em informal} agreement on the factors that influence retention (for the most part, the financial status of the student is considered to be the most important factor after the student's high-school GPA). \item However, as shown below, when we look recent experiments with student records, we find little clear support for that informal belief. In fact, we know of many universities that try to improve retention with a wide range of programs such as: \begin{itemize} \item attracting students with high performance indicators (such as their test scores) \item developing student success programs (such as first-year experience) \item or encouraging tenured-faculty to teach undergraduates \end{itemize} \end{itemize} Given the large levels of public support allocated to universities, it is important that we check the validity of these {\em informal} intuitions as well as the utility of the various retention programs such as those conducted at Kent State. This article applies data mining methods to the problem of studying student retention. Our general conclusions will be: \begin{itemize} \item It is possible to find patterns of student retention, using data mining; \item Previous data mining studies on these student records can be greatly improved using discretization, attribute selection and cross-validation over various algorithms. \end{itemize} More specifically, we will show that data mining can uncover a rich level of detail about particular universities. For example, while mining data from Kent State, we found: \begin{itemize} \item A small and specific population of students at high risk of dropping out of university. \item That the above programs (using tenured-faculty for lecturing and focusing on student performance data) is far less important than the financial status of a student. \end{itemize} Hence, we would recommend: \begin{itemize} \item Focusing more resources on the high-risk group of students, in order to improve their chances of completing a university degree. \item Discontinuing the retention programs that primarily focus on student performance indicators or that advocate using tenured-faculty for lecturing. \end{itemize} While these conclusions are specific to Kent State, the method for finding them is quite general and could be applied to other universities in order to find their most specific and most important student retention patterns. We welcome contacts from other researchers who wish to repeat our analysis on their local data. %\input{litreview} \section{Literature Review} It is no news that higher education institutions are facing the problem of student retention, which affects graduation rates as well. Colleges with higher freshmen retention rate tend to have higher graduation rates within four years. The average national retention rate is close to 55\% and in some colleges fewer than 20\% of incoming student cohort graduate \citep{Dru94}, and approximately 50\% of students entering in an engineering program leave before graduation \citep{Sca2000}. \citet{Tin82} reported national dropout rates and BA degree completions rates for the past 100 years to be constant at 45 and 52 percent respectively, except for the World War II period (see Figure \ref{figTintoCompletionRates} for the completion rates from 1880 to 1980). \citet{Till} at Valdosta State University (VSU) projected lost revenues per 10 students, who do not persist their first semester, to be \$326,811. Although gap between private institutions and public institutions in terms of first-year students returning to second year is closing, the retention rates have been constant for both types of institutions \citep[see Figure \ref{figACTRetRate}]{ACT2007}. National Center for Public Policy and Higher Education (NCPPHE) reported the U.S. average retention rate for the year 2002 to be 73.6\% \citep{NCP2007}. This problem is not only limited to the U.S. institutions, but also for the institutions in other countries such as U.K and Belgium. The U.K. national average freshmen retention for the year 1996 was 75\% \citep{Lau2003}, and \citet{Van07} found that 60\% of the first generation first-year students in Belgium fail or dropout. %\includegraphics[scale=1.0]{TintoCompletionRates} \begin{figure}[htbp] \begin{center} \includegraphics[scale=1.0]{TintoCompletionRates.pdf} \caption[BA Degree Completion Rates, 1880-1980]{BA Degree Completion Rates for the period 1880 to 1980, where Percent Completion is the Number of BAs Divided by the Number of First-time Degree Enrollment Four Years Earlier \citep{Tin82}} \label{figTintoCompletionRates} \end{center} \end{figure} \begin{figure} \begin{center} \includegraphics[scale=0.7]{FirstYearRetentionRates18Years.pdf} \end{center} \caption[Percentage of First-Year Students Who Return for Second Year]{Percentage of First-Year Students at Four-Year Colleges Who Return for Second Year \citep{ACT2007}} \label{figACTRetRate} \end{figure} Various researchers have studied this problem extensively, using theoretical models \citep{Tin75,Tin88,Spa70,Spa71,Bea80}, traditional models \citep{Ter80,Pas79,Pas80}, and data mining techniques \citep{Dru94,San95,Mas99,Ste2001,Vei2004,Bar2004,Sal2004,Sup2006,Suj2006,Her2006,Atw2006,YuD2007,Del2007}. As shown below, we can improve those prior results by augmenting standard data mining with {\em discretization}, {\em attribute selection}, and {\em cross-validation} over various algorithms. As documented in~\citet{Ada2005}, the literature on retention in higher education is extensive, and although various researchers have tested theoretical models and noted attributes critical to student retention, these theories need to be tested from time-to-time. New generation of data miners make the testing easier, and possibly can find new theories or reject old theories using state-of-the art learning algorithms. In this section, we focus on the literature relating to data mining and the student retention problem. The lesson of this section is that learning patterns of student retention is very difficult and, despite decades of effort, there is much room for improvement in the current state of the art. \subsection{Data Mining for Student Retention} \citet{Dru94} were among the first researchers to apply knowledge discovery algorithm to study the student retention problem. The authors applied TETRAD II, a casual discovery program developed at Carnegie Mellon University, to the U.S. news college ranking data to find the factors that influenced student retention, and they found that the main factor of retention was the average test score. Using linear regression, the authors found that test scores alone explained 50.5\% of the variance in freshmen retention rate. In addition, they concluded that other factors such as student-faculty ratio, faculty salary, and university's educational expense per student were not casually (directly) related to student retention; and suggested that to increase student retention universities should increase the student selectivity. \citet{San95} used 49er, a pattern discovery process developed by \citet{Zyt1993}, to find patterns in the form of regularities from student databases related to retention and graduation. The authors found that academic performance in high school was the best predictor of persistence and better performance in college, and that the high school GPA was a better predictor than the ACT composite score. In addition, they found that no amount of financial aid influenced students to enroll for more terms. \citet{Mas99} applied Markov chains modelling technique to create predictive models for the student dropout problem. By tracking the students for 15 years, the authors created state variables for the number of exams appeared, average marks obtained, and the continuation decision. Using data mining, \citet{Ste2001} studied the effects of student characteristics to persistence and success in an academic program at a community college. They found that the student's GPA, cumulative hours attempted, and cumulative hours completed were the significant predictors of persistence, and that young males were a high risk group. \citet{Vei2004} used decision trees (CHAID) to study the high school dropouts. Using 25-fold cross-validation, the overall misclassification rates of drop-outs who were classified as non-dropouts were 15.79\% and 10.36\%. In this study, GPA was the most significant predictor of persistence. \citet{Sal2004} used clustering algorithms and C4.5 to study graduate student retention at Industrial University of Santander, Colombia. The authors found that the high marks in the national pre-university test predicted a good academic performance, and that the younger students had higher probabilities of a good academic performance. \citet{Bar2004} used neural networks and Support Vector Machines (SVM) to study graduation rates; the first-year advising center (University College at University of Oklahoma) collected data via a survey given to all incoming freshman. It is worthwhile to note that \citet{Bar2004} excluded all the missing data from the study, which constituted for approximately 31\% of the total data. Overall misclassification rate was approximately 33\% for various dataset combinations. The authors used principal component analysis to reduce the number of variables from 56 to 14, however, reported that the results using the reduced datasets were ``much worse'' than the complete datasets. \citet{Sup2006} applied discriminant analysis, neural networks, random forests, and decisions trees to survey data at the University of Belgium to classify new students in low-risk, medium-risk, and high-risk categories. The authors found that the scholastic history and socio-family background were the most significant predictors of risk. The overall classification rates for decision trees, random forests, neural networks, and linear discriminant analysis were 40.63\%, 51.78\%, 51.88\%, and 57.35\% respectively. Using the National Student Clearinghouse (NSC) data, \citet{Suj2006} differentiated between stopout, retained, and transfer students. The overall classification rates for the validation sets using logistic regression, neural networks, C5.0 were 80.7\%, 84.4\%, and 82.1\% respectively. \citet{Her2006} used American College Test's (ACT) student profile section data, NSC data, and the institutional student information system data for comparing the results from the decision trees, the neural networks and logistic regression to predict retention and degree-completion time. The author substituted mean average ACT scores for missing scores. Decision trees created using C5.0 performed the best with 85\% correct classification rate for freshmen retention, 83\% correct classification rate for degree completion time (three years or less), 93\% correct classification rate for degree completion time (six years or more ) for the validation datasets. \citet{Atw2006} used University of Central Florida's student demographic and survey data to study the retention problem with the help of data mining. In this study, university retained approximately 82\% of the freshmen from the study, and it used 285 variables to create data mining models. The authors used nearest neighbor algorithm to impute more than 60\% observations with missing values. Using decision trees with the entropy split criterion, the authors obtained precision of 88\% for the not-retained outcome using the test data, and the actual retention rate for this test data set was 82.61\%. \citet{YuD2007} studied the data from Arizona State University using decision trees, and included variables, such as demographic, pre-college academic performance indicators, current curriculum, and academic achievement. Some of the important predictor variables were accumulated earned hours, in-state residence, and on campus living. To study the retention problem using data mining for the admissions data, \citet{Del2007} applied various attribute evaluation methods, such as Chi-square gain, gain ratio, and information gain, to rank the attributes. In addition, the authors tested various classifiers, such as na\"ive Bayes, AdaBoost M1, BayesNet, decision trees, and rules, and noted that AdaBoost M1 with Decision Stump classifier performed the best in terms of precision and recall, hence, used this classifier for further experimentation. The authors balanced the class variable (retained and not retained) and obtained over 60\% classification rates for both retained and not retained outcome. The authors concluded that the number of programs that the student applied to that specific institution and the student's order of program admit preference were the most significant predictors of retention. \citet{Pittman2008} compared various data mining techniques (artificial neural networks, logistic regression, Bayesian Classifiers, and decision trees) applied to the student retention problem, and also used attribute evaluators to generate rankings of important attributes. The author concluded that logistic regression performed the best in terms of ROC-curve area. \begin{sidewaystable} \begin{center} \footnotesize \begin{tabular}[t]{p{1.2in}p{1.2in}p{.4in}p{.5in}p{.5in}p{1.2in}p{.5in}p{1.6in}} {\bf Author (Year)} & {\bf Notes} & {\bf Cohort Size} & {\bf Retained (\#)} & {\bf Retained (\%)} & {\bf Measure of Accuracy} & {\bf Coeffes Used?} & {\bf Techniques Used} \\ \hline \citet{Spa71} & & 683 & 615 & 90.04\% & $R^2$ of .3132 for men and .3879 for women & Yes & Multiple regression \\ \hline \citet{Bea80} & & 906 & 769 & 84.88\% & $R^2$ of .22 for women and 0.09 for men & Yes & Multiple regression \\\hline \multirow{4}{*}{Terenzini (\citeyear{Ter80})} & study 1 & 379 & 60 & 15.80\% & $R^2$ of .246 & Yes & discriminate analyses \\ & study 3 & 518 & 428 & 82.63\% & $R^2$ of .256 & Yes & Multiple regression \\ & study 5 & 763 & 673 & 88.20\% & $R^2$ of .309 & Yes & discriminate analyses \\ & study 6 & 763 & 673 & 88.20\% & $R^2$ of .476 for men and .553 for women & Yes & discriminate analyses \\\hline \citet{Sta89} & & 323 & 294 & 91.00\% & & Yes & Logistic regression \\\hline \cite{Dey93} & & 947 & 152 & 16.00\% & Multiple R 0.354, 0.351, and 0.323 & Yes & logit, probit, and regression \\\hline \citet{Mur99} & & 8667 & 5200 & 60\% & estimated ret prob 59.3\% & Yes & Survival Analysis/ Hazard regression \\\hline \citet{Bre2002} & & 3535 & 3121 & 88.30\% & $R^2$ of 0.022 & Yes & Logistic regression \\\hline \citet{glynn2003} & any dropout; not only first-year; accuracies based on the training data & 3244 & 1592 & 49.08\% & overall accuracy of 83\% & Yes & Logistic regression \\\hline \multirow{3}{*}{\citet{Her2005}} & & 5261 & 4014 & 76.30\% & 77.4\% accuracy & Yes & Logistic regression \\ & & 4298 & 3314 & 77.10\% & & Yes & \\ & & 4671 & 4040 & 83.50\% & 85.4\% accuracy & Yes & \\\hline \multirow{3}{*}{\citet{Suj2006}} & & 2,444 & 1943 & 79.50\% & 81.6\% accuracy on training; 80.7\% on validation & Yes & Logistic regression \\ & & 2,445 & 1994 & 79.50\% & 83.9\% accuracy on training; 82.1\% on validation & & Neural Network \\ & & 2,445 & 1994 & 79.50\% & 85.5\% on training; 84.4\% on validation & & C4.5 \\\hline \citet{Her2006} & & 8,018 & 6037 & 75.29\% & accuracy close to 75\% & & Neural Networks; CHAID, C4.5, CR\&T; Logistic regression \\\hline \multirow{2}{*}{\citet{Atw2006}} & training & 3,829 & 3149 & 82.24\% & precision for drop-outs 91, 84, 84, 78 & & \multirow{2}{1.2in}{decision trees (entropy, chi-sq, gini) and logistic regression} \\ & test & 5,990 & 4,881 & 81.49\% & precision for drop-outs 88, 82,82,73 & & \\\hline \citet{Del2007} & & & & 50\% & precision varied from 57\% to 60\% & & AdaBoost M1 with Decision Stumps \\\hline \citet{Pittman2008} & & 21136 & 17139 & 81.10\% & overall accuracy of 78-81\%; not-retained precision from 44-63\% & & Logistic regression, neural network, bayes, J48 \\\hline \normalsize \end{tabular} \caption{Techniques and Accuracies Reported in Literature} \label{tabTechniquesAcc} \end{center} \end{sidewaystable} \subsection{Assessing the State of the Art}\label{sect:assess} Table \ref{tabTechniquesAcc} lists techniques used in the studied literature, where the cohort sizes were available, along with the reported performance measures. Clearly, there is much room for improvement in the current state of the art: \begin{itemize} \item It is a recommended data mining practice to divide the data into a train and test set, learn on the train set, then assess the learned theory on the test set~\citep{Wit2005}. Otherwise, if a theory is tested via the data used to build that theory, this test can over-estimate theory performance. For example, the Glynn et al. result of Table~\ref{tabTechniquesAcc} seems impressive (a 83\% accuracy on a data set with 49.08\% a retention rate); however, that result should be repeated using some {\em hold-out} test set. \item All the regression studies from 1971 to 1999 report $R^2$ values under 0.6. This $R^2$ value is a measure of how well future outcomes are likely to be predicted by the model. The maximum value of $R^2$ is one and $R^2$ values under 0.6 indicate very weak predictive abilities. \item The accuracy reports are very close to the {\em ZeroR} theoretically lower-bound on performance. ZeroR is a baseline classifier that simply returns the majority class. For example, Herzog studied a data set with a 83.5\% retention rate (see Table \ref{tabTechniquesAcc}). ZeroR, applied to this data set, would be correct in 83.5\% of cases. Therefore, the 85.4\% accuracy of Herzog's data miners is very close to the ZeroR lower-bound; i.e. the sophisticated analysis of that paper could be very nearly replicated using the dumbest of learners (ZeroR). \end{itemize} The last three results of Table \ref{tabTechniquesAcc} do not report their accuracies. However, these can be calculated in the following way. Let A, B, C, D be the true negatives, false negatives, false positives, and true positives respectively of a predictor that some student will attend some year of university. From \citet{zhang07} and \citet{me07e}, we say that (A,B,C,D) can be used in the following performance measures: {\footnotesize \begin{eqnarray} pd = recall &=& D (B+D)\label{eqpd}\\ pf = false\;alarm &=& C / (A+C) \label{eqpf}\\ prec = precision &=& D / (C+D) \label{eqprec}\\ acc = accuracy &=& (A+D) / (A+B+C+D) \label{eqaccu}\\ neg / pos &=&(A+C) / (B+D) \label{eqnegpos} \end{eqnarray}} Note that all these performance measures assess subtly different aspects of the performance of data miner: \begin{itemize} \item ``Recall'' measures how much of the target was found. \item The ``false alarm'' rate measures what fraction of non-targets triggered the learned theory. \item ``Precision'' comments on how many targets are found in the data selected by the theory. \item ``Accuracy'' comments on how many of the targets and non-targets were accurately labeled by the learned theory. \end{itemize} In an ideal result, we can obtain high recall, low false alarms, high precision, and high accuracies. However, as discussed by \citet{zhang07} and \citet{me07e}, these values are inter-related. Hence, the ideal result is not possible. These inter-relationships are shown below: \begin{equation}\footnotesize \left(prec = \frac{D}{D+C} = \frac{1}{1+\frac{C}{D}}=\frac{1}{1+\frac{neg}{pos}\cdot\frac{pf}{recall}}\right) \Rightarrow \left(pf = \frac{pos}{neg}\cdot\frac{(1-prec)}{prec}\cdot recall\right) \label{eqpfprec} \end{equation} If a publication misses a particular performance measure, it is possible to use these equations to infer the missing value. For example: \begin{eqnarray}\footnotesize D & = & recall * pos\\ C & = & pf * neg \\ A & = & C * 1/(pf - 1) \\ acc &=& (A+D) / (neg + pos) \end{eqnarray} Using these equations, we can comment that the last three results of Table \ref{tabTechniquesAcc} can be significantly improved: \begin{itemize} \item In \citet{Atw2006}, the the precision varied from 73\% to 88\%. Using our equations, we can estimate false alarm values ($pf$) ranging from 2\% to 8\% (assuming recall values of 65\% to 90\%). In our experience, it is very rare to achieve such very low false alarm rates, especially from noisy data relating to student retention. Hence, the Atwell et al., results are somewhat surprising. \item In \citet{Del2007}, the precision varied from 57\% to 60\%. From our equations, we can estimate their false alarm rates in the range of 49\% to 63\% (assuming recall values of 65\% to 90\%). Such high false alarm rates are deprecated. \item In \citet{Pittman2008}, the reported precision varied from 44\% to 63\%. Our equations comment that these values are numerically unobtainable. For $0.78 \le acc \le 0.81$, $neg=17139$ and $pos=21136-neg$, the equations only solve for $prec \le 50$. That is, half the precision values reported by Pittman need to be reviewed. \end{itemize} In summary, learning predictors for student retention is very difficult. Despite decades of work, there is considerable room for improvement in the methods used to find patterns in student retention. As we show below, such improvements are possible if we augment standard data miners with some extra pre-processors. %\input{data} \section{Data} Data used in this study were from a mid-size public university, and were extracted from the student information system on official census dates. These data consisted all first-year freshmen's demographic, academic, and financial aid information (more than 100 attributes), as of the census reporting dates (after two weeks of semester starting date). As the higher education administrators may design effective policies when the students begin their studies, it is important to note that our emphasis was on detecting patterns based only on the first-term data, and that too only beginning of the term data. We created three dependent variables: RET1, if the student returned after one year; RET2, if the student returned after two years; and RET3, if the student returned after three years. The overall distribution of these dependent variables is given in Table \ref{tabDepedentVarsDistr}. For the studied time period, the overall first-year retention rate was 71.3\%, the second-year persistence rate was 60.4\%, and the third-year persistence rate was 54.8\%. \begin{table}[ht] \begin{center} \begin{tabular}{c|rcrcrc} & \multicolumn{2}{c}{\bf{RET1}} & \multicolumn{2}{c}{\bf{RET2}} & \multicolumn{2}{c}{\bf{RET3}} \\ \cline{2-7} & Count & Percentage & Count & Percentage & Count & Percentage \\ \hline retained=\bf{Y} & 24,039 & 71.3\% & 18,055 & 60.4\% & 14,362 & 54.8\%\\ retained=\bf{N} & 9,673 & 28.7\% & 11,857 & 39.6\% & 11,854 & 45.2\%\\ \hline Total & 33,712 & 100\% & 29,912 & 100\% & 26,216 & 100\% \end{tabular} \caption{Distribution of Dependent Variables} \label{tabDepedentVarsDistr} \end{center} \end{table} In the Integrated Postsecondary Education Data System (IPEDS), for the U.S. only, degree-granting, Doctoral degree offering, 4-year and above institutes (excluding University of Phoenix-Online Campus), and cohort size greater than 3,000, we found that the full-time freshmen retention rate had a range from 59\% to 96\%, and the cohort size had a range from 3,117 to 8,025. In this list of institutions, Kent state university ranked 38 in the full-time retention percentage and 26 in the cohort size \citep{ipeds}. Thus, Kent state data are representative of other similar size universities, and the data mining approach could be generalized to other universities. \subsection{Attribute Groups} The data mining methods discussed below used {\em attribute selection} to prune measurements that are poor predictors for the target class. Therefore, our data miners can be used to assess various hypotheses relating to student retention: \begin{itemize} \item If an hypothesize claims that attributes $X,Y,Z$ are important... \item ... and if our learners prune those attributes ... \item ... then that is evidence against that hypothesis. \end{itemize} Accordingly, before applying our data miners, we take care to divide our attributes into the active hypothesis that they supprot: \begin{itemize} \item[H1:] The {\em financial aid} hypothesis. \citet{San95} found that no amount of financial aid influenced students to enroll for more terms; whereas \citet{Her2005} found that upper-income students had reduced dropout odds compared to those from middle and lower incomes. According to \citet{Joh2000}, ``the research literature remains ambiguous'' regarding the influence financial aid on recruitment and retention. \item[H2:] The {\em academic performance} hypothesis. Although there is no doubt that high school GPA and high school preparedness has a significant impact on persistence, researchers have often questioned the effects of standardized college entrance examinations (ACT/SAT). \citet{Wau94} found that SAT and ACT scores had no relationship with retention, whereas \citet{Mur99} found that SAT scores had some predictive value, although inferior compared to high school GPA. \citet{Des2002} noted that high GPA lowered the risk of dropout, but the effect diminished over time, and that the financial aid was an insignificant factor for increasing graduation, however, it indeed reduced the student stopout. In their comprehensive literature review, \citet{Lot2004} found that high school GPA had the strongest relationship with college retention in the academic factors, but ACT assessment scores had a moderate impact. \item[H3:] The {\em faculty tenure and experience} hypothesis. \citet{ehrenberg2005tenured} found that for every 10 percentage point increase in the percentage of part-time faculty and not on tenure-track full-time faculty, there was a 3-5 percentage point reduction in the institution's graduation rate. \citet{jacoby2006effects} found similar results at community colleges that increase in the ratio of part-time faculty had a negative impact on the graduation rates. \end{itemize} In the sequel, we will return to these hypotheses to comment on which were most useful for predicting student retention. Table \ref{tabAttrswHypothesis} lists attributes that we grouped together under each hypothesis. %\input{attrsbyHypothesis} \begin{sidewaystable} \begin{center} \scriptsize \begin{tabular}{|p{1in}p{1.6in}|p{0.7in}p{1.8in}|p{0.85in}p{2.2in}|} \hline {\bf Category} & Financial Aid & {\bf Category} & Performance Indicators& {\bf Category} & Faculty Type \& Experience\\ \hline {\bf Attribute} & {\bf Description} & {\bf Attribute} & {\bf Description} & {\bf Attribute} & {\bf Description} \\ \hline FinAidAwardType\_G & Financial aid amount of grants & ACT\_COMP & ACT comprehensive score (old) & FacExpLT1Cnt & Count of courses taught by faculty [CCTF] w/ less than 1 yr experience \\ FinAidAwardType\_J & Financial aid amount of jobs & ACT\_ENGL & ACT english score (old) & FacExpLT1Ratio & Ratio of courses taught by faculty [RCTF] w/ less than 1 yr experience to the total courses \\ FinAidAwardType\_L & Financial aid amount of loans & ACT\_MATH & ACT math score (old) & FacExp1to5Cnt & CCTF w/ experience between 1 \& 5 \\ FinAidAwardType\_S & Financial aid amount of scholarship & ACT1\_COMP & ACT comprehensive score (new) & FacExp1to5Ratio & RCTF w/ experience between 1 \& 5 to the total courses \\ FinAidAwardType\_W & Financial aid amount of waiver & ACT1\_ENGL & ACT english score (new) & FacExp6to10Cnt & CCTF w/ experiencAe between 6 \& 10 \\ FinAidDEPENDENCY & Dependency status & ACT1\_MATH & ACT math score (new) & FacExp6to10Ratio & RCTF w/ experience between 6 \& 10 to the total courses \\ FinAidFATHER\_ED & Father's education level & ACTEQUIV & ACT equivalent of the sat score & FacExp11to15Cnt & CCTF w/ experience between 11 \& 15 \\ FinAidFATHER\_WAG & Father's income & MaxACT & Max of ACT score and ACT equivalent & FacExp11to15Ratio & RCTF w/ experience between 11 \& 15 to the total courses \\ FinAidMOTHER\_ED & Mother's education level & COMP\_READ & Compass read score & FacExp16to20Cnt & CCTF w/ experience between 16 \& 20 \\ FinAidMOTHER\_WAG & Mother's income & COMP\_WRITE & Compass write score & FacExp16to20Ratio & RCTF w/ experience between 16 \& 20 to the total courses \\ FinAidOfferedInd & Financial aid offered indicator & SAT\_TOT & SAT total score & FacExp21to25Cnt & CCTF w/ experience between 21 \& 25 \\ FinAidPARENT\_AGI & Parent's adjusted gross income & SAT\_VERB & SAT verbal score & FacExp21to25Ratio & RCTF w/ experience between 21 \& 25 to the total courses \\ FinAidPARENT\_HOU & Parents's household size & HS\_CODE & High school code & FacExp25to30Cnt & CCTF w/ experience between 25 \& 30 \\ FinAidPARENT\_MAR & Parents's martial status & HS\_GPA & High school gpa & FacExp25to30Ratio & RCTF w/ experience between 25 \& 30 to the total courses \\ FinAidPARENT\_TAX & Parents's tax form type & HS\_PERCENT & High school percentile & FacExpGT31Cnt & CCTF w/ experience greater than 31 years \\ FinAidSPOUSE\_WAG & Spouse's wages & HS\_RANK & High school rank & FacExpGT31Ratio & RCTF w/ experience greater than 31 years to the total courses \\ FinAidSTUDENT\_AG & Student's adjusted gross income & HS\_SIZE & High school class size & NoTenureFacCnt & Count of courses taught by no rank faculty \\ FinAidSTUDENT\_HO & Student's household size & RankHSGPA & Percentile of hs gpa among all freshmen & NoTenureFacRatio & Ratio of courses taught by no rank faculty to the total courses \\ FinAidSTUDENT\_MA & Student's martial status & RankMaxACT & Percentile of max ACT among all freshmen & NTTFacCnt & Count of courses taught by ntt faculty \\ FinAidSTUDENT\_TA & Student's tax form type & ANTH18 & Enrolled in anthropology course & NTTFacRatio & Ratio of courses taught by ntt faculty to the total courses \\ FinAidSTUDENT\_WA & Student's wage & BSCI10 & Enrolled in biological science course & TTFacCnt & Count of courses taught by tenured/tenure-track faculty \\ FirstGenInd & First generation indicator & CHEM10 & Enrolled in chemistry course & TTFacRatio & Ratio of courses taught by tenured/tenure-track faculty to the total courses \\ TotalFinAidOffered & Total financial aid offered & ENG10 & Enrolled in English course & & \\ & & ENG11 & Enrolled in English course & & \\ & & GEOL11 & Enrolled in geology course & & \\ & & LEST16 & Enrolled in leisure studies course & & \\ & & MATH10 & Enrolled in math 100 level course & & \\ & & MATH11 & Enrolled in math 110 level course & & \\ & & MATH12 & Enrolled in math 120 level course & & \\ & & MATH14 & Enrolled in math 14 level course & & \\ & & PHY11 & Enrolled in physics 11 level course & & \\ & & PEP15 & Enrolled in physical ed 15 level course & & \\ \hline \end{tabular} \label{tabAttrswHypothesis} \caption{List of Attributes by Stated Hypotheses} \end{center} \end{sidewaystable} \normalsize %\input{experiment} \section{Building the Experiment} In Section~\ref{sect:assess}, we assert that a good data miner should do better than the simplistic ZeroR learner. Table \ref{tabDepedentVarsDistr} tell us that that lower bound is: \begin{itemize} \item For first year retention: 71.3\%. \item For second year retention: 60.4\%. \item For third year retention: 54.8\%. \end{itemize} As discussed below, we will be able to do much better than some, but not all, of these targets. This was achieved by \begin{itemize} \item Removing spurious attributes using {\em feature subset selection}; \item Exploring {\em a large range of classifiers}; \item Assessing the learned theories by their {\em variance}, as well as their {\em median} performance. \item Assessing the learned theories by their {\em variance}, as well as their {\em median} performance. \item Study the {\em delta} of student factors {\em between} those who stay and those who are retained. \end{itemize} \subsection{Feature Subset Selection} Table \ref{tabAttrswHypothesis} shows a sample of the 103 attributes used in this study. Our pre-experimental suspicion was that some of the attributes were ``noisy''; i.e. contain signals not related to the target of prediction retention. Therefore, before we learn a theory, we first explored {\em attribute selection}. Note that the number of attributes to select is crucial in the analysis of the data, because it allows us to comment on the hypotheses shown in the last section. If removal of attributes from an hypothesis does not change the performance of the prediction, then that hypothesis is spurious. In this experiment, we ranked the 103 attributes from most informative to least informative. We then built theories using the top $n\in\{5,10,..,100,103\}$ ranked attributes. Attributes were then discarded if adding them in did not improve the performance of our retention predictors. The attributes were ranked using one of four methods: CFS, Information Gain, chi-squared, and One-R. {\em Correlation-based feature selection} constructs a matrix of feature to feature, and feature-to-class correlations~\citep{Hall00correlation-basedfeature}. CFS uses a best first search by expanding the best subsets until no improvement is made, in which case the search falls to the unexpanded subset having the next best evaluation until a subset expansion limit is met. {\em Information Gain} uses an information theory concept called $entropy$. Entropy measures the amount of uncertainty, or randomness, that is associated with a random variable. Thus, high entropy can be seen as a lack of purity in the data. Information gain, as described in~\cite{Mitchell97} is an expected reduction of the entropy measure that occurs when splitting examples in the data using a particular attribute. Therefore an attribute that has a high purity (high information gain) is better at describing the data than the one that has a low purity. The resulting attributes are then ranked by sorted their information gain scores in a descending order. The {\em chi-squared} statistic is used in statistical tests to determine how distributions of variables are different from one another~\citep{Moore06}. Note that these variables must be categorical in nature. Thus, the chi-squared statistic can evaluate an attribute's worth by calculating the value of this statistic with respect to a class. Attributes can then be ranked based on this statistic. The {\em One-R} classifier, described below, can be used to deliver top-ranking attributes. One-R constructs and scores rules using one attribute. Feature selectors using One-R sort the attributes based on these scores. \subsection{Classifiers} In data mining, classifiers are used to learn connections between independent features and the dependent feature (called the {\em class}). Once these patterns are learned, we can predict outcomes in new data by reflecting on data that has already been examined. This study tried six different classifiers: One-R, C4.5, ADTrees, Naive Bayes, Bayes networks, and radial bias networks. These are some of the well-known and standard classifiers in the machine learning field, except for ADTrees. {\em One-R}, described in~\cite{holte93}, builds rules from the data by iteratively examining each value of an attribute and counting the frequency of each class for that attribute-value pair. An attribute-value is then assigned as the most frequently occurring class. Error rates of each of the rules are then calculated, and the best rules are ranked based on the lowest error rates. A {\em radial basis function network} (RBFN) is an artificial neural network (ANN) that utilizes a radial basis function as an activation function~\citep{rbfnIntroBors}. An ANN's activation function is used in order to offer non-linearity to the network. This is important for multi-layer networks containing many hidden layers, because their advantages lie in their ability to learn on non-linearly separable examples. {\em C4.5}~\citep{quinlanC4.5} is an extension to the ID3~\citep{quinlanID3} algorithm. A decision tree (shown in Figure~\ref{fig:decisionTree}) is constructed by first determining the best attribute to make as the root node of the tree~\citep{Mitchell97}. ID3 decides this root attribute by using one that best classifies training examples based upon the attribute's information gain (described above)~\citep{quinlanID3}. Then, for each value of the attribute representing any node in the tree, the algorithm recursively builds child nodes based on how well another attribute from the data describes that specific branch of its parent node. The learning stops when the tree perfectly classifies all training examples, or when all attributes used. C4.5 extends ID3 by making several improvements, such as the ability to operate on both continuous as well as discrete attributes, training data that contains missing values for a given attribute(s), and employ pruning techniques on the resulting tree. \begin{figure} \begin{center} \includegraphics[width=6.5in]{Investment_decision_tree.pdf} \end{center} \caption{A decision tree consists of a root node and descending children nodes who denote decisions to make in the tree's structure. This tree, for example, was constructed in an attempt to optimize investment portfolios by minimizing budgets and maximizing pay-offs. The top-most branch represents the best selection in this example.} \label{fig:decisionTree} \end{figure} {\em ADTrees} are decision trees that contain both decision nodes, as well as prediction nodes~\citep{Freund99thealternating}. Decision nodes specify a condition, while prediction nodes contain only a number. Thus, as an example in the data follows paths in the ADTree, it only traverses branches whose decision nodes are true. The example is then classified by summing all prediction nodes that are encountered in this traversal. ADTrees, however, differ from binary classification trees, such as C4.5, where those trees only traverses a single path down the tree. \begin{figure} \begin{center} \includegraphics[width=4.5in]{bayesnet_diagram.pdf} \end{center} \caption{In this simple Bayesian network, the variable {\em Sprinkler} is dependent upon whether or not it's raining; the sprinkler is generally not turned on when it's raining. However, either event is able to cause the grass to become wet - if it's raining, or if the sprinkler is caused to turn on. Thus, Bayesian networks excel at investigating information relating to relationships between variables.} \label{fig:bayesnetwork} \end{figure} A {\em naive Bayes} classifier uses Bayes' theorem to classify training data. Bayes' theorem, as shown in Equation~\ref{eq:bayesTheorem}, determines the probability $P$ of an event $H$ occurring given an amount of evidence $E$. This classifier assumes feature independence; the algorithm examines features independently to contribute to probabilities, as opposed to the assumption that features depend on other features. Surprisingly, even though feature independence is an integral part of the classifier, it often outperforms many other learners ~\citep{NB-performance,domingos97optimality}. \begin{equation} Pr(H|E) = \frac{Pr(E|H) * Pr(H)}{Pr(E)} \label{eq:bayesTheorem} \end{equation} {\em Bayesian networks}, illustrated in Figure ~\ref{fig:bayesnetwork}, are graphical models that use a directed acyclic graph (DAG) to represent probabilistic relationships between variables. As stated in~\cite{Heckerman96atutorial}, Bayesian networks have four important elements to offer: \begin{enumerate} \item{Incomplete data sets can be handled well by Bayesian networks. Because the networks encode a correlation between input variables, if an input is not observed, in will not necessarily produce inaccurate predictions, as would other methods.} \item{Causal relationships can be learned about via Bayesian networks. For instance, we can find whether a certain action taken would produce a specific result and to what degree.} \item{Bayesian networks promote the amalgamation of data and domain knowledge by allowing for a straightforward encoding of causal prior knowledge, as well as the ability to encode causal relationship strength.} \item{Bayesian networks avoid over fitting of data, as ``smoothing'' can be used in a way such that all data that is available can be used for training.} \end{enumerate} \subsection{Cross-Validation} The value of different attributes can be assessed using equations one to four. If we use multiple {\em hold out} test sets, we can also discover the variance in these performance figures. In this experiment, we performed a 5 $\times$ 5 cross-validation i.e. we partitioned the data five times into a testing set consisting of $\frac{1}{5}$-th of the data and a training set of $\frac{4}{5}$-ths of the data. After the five rounds, we recorded the median values of recall and false alarm rates. \subsection{Contrast Set Learning} After determining the subset of the attributes that best predict for student retention, we conducted a {\em contrast set study}. Contrast set learners like TAR3~\citep{me07} seek attribute ranges that are most {\em different} in various outcomes. One way to read these contrast sets are as {\em treatments} that promise if action X was applied to a domain, then this would favor outcome X over outcome Y. In our case, we used TAR3 in two ways: \begin{itemize} \item Firstly, we will use TAR3 to find which treatments most select for retention; \item Secondly, we will run TAR3 in the opposite direction to find the treatments that most select for students leaving university. \end{itemize}. In the first case, TAR3 is being used to find actions that most encourage retention. In the second case, TAR3 is being used to find the worst possible actions that most increase the probability of a student leaving. \section{Analysis of Experimental Results} \subsection{Evaluation Metrics} The evaluation metrics used in this experiment are standard data mining performance measures of a method. They are: \begin{itemize} \item Probability of detection (PD); \item Probability of false alarm (PF); \item And variance PD and PF seen over the our cross-validation study. \end{itemize} Variance in these values provides insight into how much reliability a classifier supports on the data. For example, if a method's PD values ranges from very low to very high, we can conclude that the particular method is inconsistent in its probabilities of detection. For our studies, we rejected anything with a variance greater than $\pm 25\%$. The above statistics were collected over 1500 experiments, which were repeated 20 times (to check for conclusion stability). In all, we conducted \[ 5*5*4*6*3*20=36,000\] experiments; i.e. 5 $\times$ 5 cross-validation using four feature subset selectors and 6 different learners, for the 3 data sets of Section~3 (recall from \S3 that those three data sets contained data about first, second, and third year retention). This was repeated 20 times using the top $n\in {5,10,15,..,100,103}$ attributes as found by the feature selector. \subsection{First Results} After rejecting all results with (1)~a PD lower than the ZeroR limit; (2)~a PD variance greater than $\pm 25\%$; and (3)~a PF higher than 25\%, we found that we had no predictors for Year1 or Year2 retention. This is the first major finding for this research: {\em it is very difficult to predict lower year retention}. Note that this result is consistent with prior results discussed above in our literature review. For the rest of this study, we will focus only on third year retention. The case for focusing on third year retention is quite clear: \begin{itemize} \item If the goal is to provide a complete university education for a student, then predicting survival till second year is less interesting than lasting till third year. \item Third year retention implies second and first year retention. \end{itemize} \subsection{Ranking with the Mann-Whitney Test} After pruning results with low PD, high PF, or high PD variance, we ranked the remaining results via a Mann-Whitney test (95\% confidence). We determined the ranks by counting how many times a combination won compared to another combinations. The method that won the most number of times was then given the highest rank. The table in Figure~\ref{fig:ranktable} shows the top ten ranking combinations based on a PD performance measure. Note: we gave identical ranks to those treatments whose win value was equal in magnitude. \begin{table} \begin{center} \begin{tabular}{| c | c | c | c |} \hline Rank & Number of Attributes & FSS & Classifier \\ \hline 61& 30&oneR&bnet \\ 61&50&cfs&adtree \\ 57& 50&oneR&adtree \\ 56& 30&oneR&adtree \\ 55& 30&cfs&adtree \\ 52& 50&oneR&bnet \\ 51& 30&infogain&adtree \\ 51& 30&cfs&bnet \\ 48& 50&infogain&adtree \\ \end{tabular} \end{center} \caption{The top ten ranking treatments for third year retention. Ranks represent how many times a particular treatment wins over all other treatments in the experiment.} \label{fig:ranktable} \end{table} Since similar results were achieved using 30 or 50 attributes, we applied Occam's Razor and focused on the 30 attributes found to be best for oneR/bnet. For these 30 attributes, we studied all their ranges. Figure~\ref{fig:ranktable} shows the ranges which, in isolation, select for retention at a probability greater than the ZeroR limit for (for third year, that ZeroR limit is 55\%). In terms of assessing different hypothesis, the third column of Figure~\ref{fig:ranktable} is most informative: \begin{itemize} \item The ranges shown at the top of the table are most predictive for third year retention. Note the dominance of ``Financial Aid'' attributes from Figure~\ref{fig:rankingsr}. \item Attributes related to student ``Performance'' are rarer. \item None of the attribute ranges include the ``Faculty Type and Experience'' attributes of Figure~\ref{fig:rankingsr}. \end{itemize} From this analysis, we made two tentative conclusions: \begin{itemize} \item Using experienced faculty-level instructors is {\em not} predictive for third year retention. \item Issues relating to financial aid dominate over student performance. \end{itemize} \subsection{Ranking with Contrast Set Learning} The counter-case to this conclusion might be that Figure~\ref{fig:ranktable} only discusses the effect of attribute ranges in {\em isolation}. It is possible that combination of factors might lead to different conclusions. The TAR3 treatment learner was used to test this possibility. We let TAR3 build rules of up to size 10 (i.e. ten combinations of attribute ranges) from the 30 attributes selected by the best learning combination of Figure~\ref{fig:ranktable}. It turned out that this max size of 10 ranges was much larger than necessary: TAR3 never found combinations larger than three ranges. %\input{bestrangessorted} \begin{table}[!t] \footnotesize \begin{tabular}{c|c|c|p{0.7in}|lr} \#& $P(Ret3|X)$ &Support&&\multicolumn{2}{c}{X (Feature = Range)}\\\cline{5-6} & (percent) & (\#students) & Hypothesis & Feature & Range\\\hline 1 & 79 & 1,752 & Financial Aid & Student's Wage & 7850 to 9958 \\ 2 & 73 & 3,751 & Financial Aid & Parent's Adjusted Gross Income & 96636 to inf \\ 3 & 71 & 4,152 & Financial Aid & Student's Adjusted Gross Income & 4830 to 7916 \\ 4 & 71 & 3,148 & Financial Aid & Mother's Income & 42957 to inf \\ 5 & 70 & 5,873 & Financial Aid & Father's Income & 52366 to inf \\ 6 & 69 & 5,622 & Financial Aid & Student's Wage & 4093 to 7851 \\ 7 & 69 & 5,838 & Performance & High School Percentile & 81 to inf \\ 8 & 68 & 2,523 & Financial Aid & Student's Dependency Status & I \\ 9 & 68 & 7,502 & Financial Aid & Father's Education Level & 3 \\ 10 & 67 & 6,045 & Financial Aid & Parent's Adjusted Gross Income & 58551 to 96636 \\ 11 & 66 & 12,215 & Financial Aid & Student's Tax Form & 2 \\ 12 & 66 & 7,710 & Financial Aid & Mother's Education Level & 3 \\ 13 & 66 & 2,057 & Financial Aid & Student's Wage & 1.5 to 1000 \\ 14 & 66 & 10,370 & Financial Aid & First Generation Student & N \\ 15 & 65 & 7,082 & Performance & ACT Math Score (new) & 23 to inf \\ 16 & 65 & 2,780 & Financial Aid & Student's Adjusted Gross Income & 3336 to 4830 \\ 17 & 65 & 4,676 & Performance & ACT English Score (new) & 25 to inf \\ 18 & 65 & 5,669 & Performance & ACT Comprehensive Score (new) & 24 to inf \\ 19 & 65 & 13,101 & Financial Aid & Parent's Tax Form & 1 \\ 20 & 65 & 11,328 & Financial Aid & Parent's Marital Status & M \\ 21 & 64 & 6,658 & Performance & Percentile Of Max ACT Among Freshmen & 71 to inf \\ 22 & 64 & 6,952 & Performance & Max Of ACT Score And ACT Equivalent & 24 to inf \\ 23 & 64 & 2,697 & Financial Aid & Student's Tax Form & 1 \\ 24 & 63 & 17,254 & Financial Aid & Student's Marital Status & U \\ 25 & 63 & 13,063 & Financial Aid & Mother's Wages & -inf to 42957 \\ 26 & 63 & 3,126 & Financial Aid & Parent's Tax Form & 2 \\ 27 & 63 & 1,022 & Financial Aid & Student's Adjusted Gross Income & 16714 to inf \\ 28 & 62 & 15,154 & Financial Aid & Dependency & D \\ 29 & 62 & 9,459 & Financial Aid & Father's Income & -inf to 52366 \\ 30 & 61 & 8,792 & Financial Aid & Mother's Education Level & 2 \\ 31 & 61 & 8,461 & Financial Aid & Father's Education Level & 2 \\ 32 & 61 & 4,176 & Financial Aid & Student's Wage & 1904 to 4093 \\ 33 & 60 & 14,523 & & Total Enrolled Hours & 15 to 19 \\ 34 & 60 & 2,540 & Financial Aid & Student's Adjusted Gross Income & 1895 to 3336 \\ 35 & 59 & 3,780 & Performance & Compass Writing Score & -inf to 10 \\ 36 & 59 & 8,271 & Performance & ACT English Score (new) & 20 to 25 \\ 37 & 59 & 7,311 & & FirstGenInd & Y \\ 38 & 59 & 6,980 & Performance & HS\_PERCENT & 61 to 81 \\ 39 & 59 & 15,021 & Performance & Total Number of Enrolled Classes & 6 to inf \\ 40 & 59 & 5,598 & Financial Aid & Parent's Adjusted Gross Income & 1838 to 58551 \\ 41 & 58 & 3,127 & Financial Aid & Parent's Marital Status & S \\ 42 & 58 & 8,667 & Performance & ACT Composite & 20 to 24 \\ 43 & 58 & 4,767 & Performance & ACT Math Score (new) & 20 to 23 \\ 44 & 58 & 13,887 & Performance & Compass Writing Score & 74 to inf \\ 45 & 58 & 5,769 & Performance & High School GPA & 3.02 to 3.4 \\ 46 & 58 & 10,281 & Performance & RankMaxACT & 31 to 71 \\ 47 & 58 & 10,044 & Performance & MaxACT & 20 to 24 \\ 48 & 57 & 20,087 & & On-Campus Indicator & Y \\ 49 & 57 & 11,36 & Financial Aid & Father's Education Level & 4 \\ 50 & 56 & 24,826 & & Age of Student at Matriculation & -inf to 19.5 \\ 51 & 56 & 24,407 & Performance & Enrolled in English Courses & N \\ 52 & 56 & 3,660 & Performance & Percentile Of Hs Gpa Among Freshmen & 46 to 60 \end{tabular} \caption{Ranking all attribute ranges which, in isolation, predict for third year retention at a probability higher than the ZeroR limit (55\%). From the above, the strongest predictor for third year retention is a student's wage (at 79\%). On the other hand, the bottom line of this table says that the percentile of a student amongst their Freshmen cohort is little better than ZeroR (at 56\%). }\label{fig:rankingsr} \end{table} %\input{results} \section{Results} Figure \ref{fig:rankingsr} lists the rankings of all attribute ranges which, in isolation, predict for third year retention at a probability higher than the ZeroR limit (55\%), and are supported by good number of records. The top six attributes affecting third-year retention were from the financial aid hypothesis: student's wages, parent's adjusted gross income, student's adjusted gross income, mother's income, father's income, and high school percentile. Of those students who reported their wages, students who made between 7,850 and 9,958 had a 79\% retention. Similar rules were found for parent's income and adjusted gross income. It means that the students with stronger financial support usually stay in college than the students with weaker financial support. After these top six attributes, high school percentile of 81 or greater was an important attribute with 69\% of students returning after three years. Some other ``performance'' attributes were ACT scores and ranks. This supports the argument that scores do have some predictability of student retention. TAR3 results, given in Figure \ref{fig:rx}, produced simple theories (treatments) that combined ranges of various attributes that maximized the student retention. For example, the student retention was very high for students with the AGI in the range from \$7,000 to \$724,724 and father's wages were in the range from \$56,289 to \$999,999. One more interesting theory that predicted high retention was where father's education level was 3 (college) and student's rank amongst the freshmen cohort was between 66.3 and 98.4. Treatments that predicted student drop-out were based on the total number of classes student was enrolled, English 10000, an introductory college writing and supplemental instruction class, and on-campus living. Students who took less than five class, enrolled in the English 10000 class, and did not live on-campus were at high risk of drop-out. Chart on the bottom of Figure \ref{fig:rx} shows the retention percentage of each treatment. For example, students enrolled in English 10000 had a 40\% retention in their third year. Key findings were: \begin{itemize} \item Student's and parent's income capacity and levels affected student retention. Third-year retention was higher for the students with high income than the students with low income. According to treatment 1, approximately 82\% of students who had at least \$7,000 AGI and their fathers' income was at least \$56,289 returned after three years. Similarly, according to treatment 5, approximately 79\% of students who made at least \$5,383 and their parents' AGI was at least \$87,744 returned after three years. \item Students with better high school performance amongst their peers had higher chances of retention. According to treatment 2, approximately 81\% of students who had at least \$7,000 AGI and had high school percentile of 72 and better returned after three years. Approximately 79\% students who had at least 3.34 HS GPA and whose parents had an AGI of at least \$84,744 stayed after three years, given in treatment 4. \item ACT scores, rank of these scores amongst peers, and COMPASS scores affected student retention. Students with higher scores and rank had higher chances of retention. According to treatment 3, approximately 80\% of students who had at least \$7,000 AGI and had ACT math score of 21 or better returned after three years. Similarly, 77\% of students who had at least 23 in ACT composite (or SAT equivalent) and had an income of at least \$5,383 and less than \$561,500 returned after three years, given in treatment 6. \item Parent's education level had a positive effect on student retention. Students whose parents did not attend college had a lower retention compared to students whose parents did attend college. As given in treatments 7 and 10, a student was highly likely (77\%) to return after three years:~(7) if the mother of that student attended college, the student had a ACT composite score of 22 or better, the parents' AGI was at least \$84,744; ~(10) if the father of that student attended college and the student's percentile rank amongst other freshmen in the cohort was at least 66.3. \item Enrolling in fewer classes (less than five), enrolling in English 10000 (an introductory college writing class), and living off-campus had a negative effect on student retention, as given in treatments 11, 12, and 13. It is important to note that enrolling in that English course itself is not a predictor of non-retention, but the sample of the students that attended this class were at high-risk of dropping out. Given funding for further investigation, we would focus more data collection on this high-risk group. \end{itemize} \begin{figure} \small \begin{center} \begin{tabular}{r@{~}|p{5in}} \# & Treatment \\\hline 1 & 7000 $\le$ FinAidSTUDENT\_AG $<$ 724,724 and 56,289 $\le$ FinAidFATHER\_WAG $<$ 999,999 \\ 2 & 7,000 $\le$ FinAidSTUDENT\_AG $<$ 724,724 and HS\_PERCENT $\ge$ 72 \\ 3 & 7,000 $\le$ FinAidSTUDENT\_AG $<$ 724,724 and 21 $\le$ ACT1\_MATH $<$ 36 \\ 4 & 84,744 $\le$ FinAidPARENT\_AGI $<$ 999,999 and HS\_GPA $\ge$ 3.34 \\ 5 & 84,744 $\le$ FinAidPARENT\_AGI $<$ 999,999 and 5383 $\le$ FinAidSTUDENT\_WA $<$ 561,500 \\ 6 & 23 $\le$ MaxACT $<$ 35 and 5383 $\le$ FinAidSTUDENT\_WA $<$ 561,500 \\ 7 & 22 $\le$ ACT1\_COMP $<$ 35 and 84,744 $\le$ FinAidPARENT\_AGI $<$ 999,999 and\newline FinAidMOTHER\_ED=3\\ 8 & 5383 $\le$ FinAidSTUDENT\_WA $<$ 561,500 and 21 $\le$ ACT1\_MATH $<$ 36\\ 9 & HS\_GPA $\ge$ 3.34 and 32,570 $\le$ FinAidMOTHER\_WAG $<$ 533,395 \\ 10 & FinAidFATHER\_ED=3 and 66.3 $\le$ PercentileRankHSGPA $<$ 98.4\\\hline 11 & 1 $\le$ TotalClass$\le$ 5\\ 12 & ENG10=Y \\ 13 & LIVE.ON.CAMP=N \end{tabular} \includegraphics[width=3in]{gploty.pdf} \end{center} \caption{ Treatments 1 to 10 are the top ten treatments found by this analysis that increases the third year retention rates. Treatments 11,12,13 are the worst three treatments found by this analysis that {\em most decrease} the third year retention rates. The effects of each treatment, is shown on the bottom plot. }\label{fig:rx} \end{figure} \subsection{Strategic Actions} This study provides insights in student retention domain using beginning of term data. These insights can be used to design effective policies and strategic actions, such as: \begin{itemize} \item Most of the attributes were related to socio-economic levels and capacities of students and their parents; however, this cannot be controlled while admitting students, but better support programs and calculated financial-aid packaging for students with lower economic capacities can be created. \item First-year students should be encouraged to live on-campus by providing some incentives, as on-campus students have higher chances of retention. \item Special guidance and supplemental instruction in writing and reading should be provided to first-generation students. Parents of first-year generation students have considerably low-incomes than the parents of non-first-generation students, and according to the results of this study, income of parents is a critical factor in student retention even if the students had similar academic performance. \item Students are placed in the supplemental instruction classes, such as English 10000, based on their COMPASS and ACT scores. As these students' scores indicated lack of academic preparedness in some areas, academic advisers correctly place students in such classes; however, if the students fail or perform poorly in such classes, it leaves a lasting impression and sets the students to for future drop-out, even after three years. Therefore, it is paramount that advisers not only place students in supplemental instruction classes, but also ensure the success of students in these classes and improve the skills that students lack. Out of all classes considered in this study, English seemed to have the greatest impact. Intuitive as it may be, to succeed in college, students need good writing and reading skills. \end{itemize} %\input{conclusions} \section{Conclusion} Although our techniques could not predict first or second year retention with significantly higher accuracies than the baseline, these techniques obtained probability of detection approximately 15\% higher for the class value of $Y$ and 20\% higher for the class value of $N$ than the baseline percentages for third-year retention, based on the first-year beginning of the term data. In the studied literature, we have not found any studies with such a significant improvement over the baseline for the third-year retention. In addition, if policies are designed to improve third-year retention rate (using this predictive model), not only will they improve first and second year retention rates, but also the six-year graduation rates. For the studied institution, family background and family's social-economic status are critical for student's third-year persistence. Using feature subset selection methods, we found that the attributes from the ``financial aid'' hypothesis were selected the most as predictors of retention, and although the attributes from the ``performance'' hypothesis were selected, their predictability, in isolation, was lesser than the attributes from the ``financial aid'' hypothesis. None of the attributes from the ``faculty tenure and experience'' were selected by the feature subset selectors. These results could very well be true only for the studied institution; however, if the approach detailed in this study is followed, other institutions can find top performing classifier and important attributes. We recommend: ~(a) data discretization; ~(b) feature subset selection with cross-validation and evaluation the performance over various learners; ~(c) treatment learners, such as TAR3 to find succinct strategic actions in complex data. We welcome the opportunity to study data from other institutions and willing to share the experiment platform used in this study. % *************** Bibliography *************** \bibliography{paperrefs} \bibliographystyle{model5-names.bst} \begin{thebibliography}{59} \expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi \providecommand{\bibinfo}[2]{#2} \ifx\xfnm\relax \def\xfnm[#1]{\unskip,\space#1}\fi %Type = Misc \bibitem[{{ACT}(2007)}]{ACT2007} \bibinfo{author}{{ACT}} (\bibinfo{year}{2007}). \newblock \bibinfo{title}{{ACT National Collegiate Retention and Persistence to Degree Rates}}. \newblock \bibinfo{note}{\url{http://www.act.org/research/policymakers/reports/retain.% html}}. %Type = Article \bibitem[{Adam \& Gaither(2005)}]{Ada2005} \bibinfo{author}{Adam, A.~J.}, \& \bibinfo{author}{Gaither, G.~H.} (\bibinfo{year}{2005}). \newblock \bibinfo{title}{Retention in higher education: A selective resource guide}. \newblock {\it \bibinfo{journal}{New Directions for Institutional Research}\/}, {\it \bibinfo{volume}{2005}\/}, \bibinfo{pages}{107--122}. %Type = Inproceedings \bibitem[{Atwell et~al.(2006)Atwell, Ding, Ehasz, Johnson \& Wang}]{Atw2006} \bibinfo{author}{Atwell, R.~H.}, \bibinfo{author}{Ding, W.}, \bibinfo{author}{Ehasz, M.}, \bibinfo{author}{Johnson, S.}, \& \bibinfo{author}{Wang, M.} (\bibinfo{year}{2006}). \newblock \bibinfo{title}{Using data mining techniques to predict student development and retention}. \newblock In {\it \bibinfo{booktitle}{Proceedings of the National Symposium on Student Retention}\/}. %Type = Article \bibitem[{Barker et~al.(2004)Barker, Trafalis \& Rhoads}]{Bar2004} \bibinfo{author}{Barker, K.}, \bibinfo{author}{Trafalis, T.}, \& \bibinfo{author}{Rhoads, T.~R.} (\bibinfo{year}{2004}). \newblock \bibinfo{title}{Learning from student data}. \newblock {\it \bibinfo{journal}{Systems and Information Engineering Design Symposium}\/}, (pp. \bibinfo{pages}{79--86}). %Type = Article \bibitem[{Bean(1980)}]{Bea80} \bibinfo{author}{Bean, J.~P.} (\bibinfo{year}{1980}). \newblock \bibinfo{title}{Dropouts and turnover: The synthesis and test of a causal model of student attrition}. \newblock {\it \bibinfo{journal}{Research in Higher Education}\/}, {\it \bibinfo{volume}{12}\/}, \bibinfo{pages}{155--187}. %Type = Inproceedings \bibitem[{Bors(2001)}]{rbfnIntroBors} \bibinfo{author}{Bors, A.} (\bibinfo{year}{2001}). \newblock \bibinfo{title}{{Introduction of the radial basis function (rbf) networks}}. \newblock In {\it \bibinfo{booktitle}{Online Symposium for Electronics Engineers}\/} (pp. \bibinfo{pages}{1--7}). \newblock volume~\bibinfo{volume}{1}. %Type = Article \bibitem[{Bresciani \& Carson(2002)}]{Bre2002} \bibinfo{author}{Bresciani, M.~J.}, \& \bibinfo{author}{Carson, L.} (\bibinfo{year}{2002}). \newblock \bibinfo{title}{A study of undergraduate persistence by unmet need and percentage of gift aid}. \newblock {\it \bibinfo{journal}{NASPA Journal}\/}, {\it \bibinfo{volume}{40}\/}, \bibinfo{pages}{104--123}. %Type = Inproceedings \bibitem[{DeLong et~al.(2007)DeLong, Radcliffe \& Gorny}]{Del2007} \bibinfo{author}{DeLong, C.}, \bibinfo{author}{Radcliffe, P.~M.}, \& \bibinfo{author}{Gorny, L.~S.} (\bibinfo{year}{2007}). \newblock \bibinfo{title}{Recruiting for retention: Using data mining and machine learning to leverage the admissions process for improved freshman retention}. \newblock In {\it \bibinfo{booktitle}{Proceedings of the National Symposium on Student Retention}\/}. %Type = Misc \bibitem[{{Department~of~Education}(2010)}]{ipeds} \bibinfo{author}{{Department~of~Education}} (\bibinfo{year}{2010}). \newblock \bibinfo{title}{{Integrated Postsecondary Education Data System (IPEDS)}}. \newblock \bibinfo{note}{\url{http://nces.ed.gov/ipeds/datacenter/}}. %Type = Article \bibitem[{DesJardins et~al.(2002)DesJardins, Ahlburg \& McCall}]{Des2002} \bibinfo{author}{DesJardins, S.~L.}, \bibinfo{author}{Ahlburg, D.~A.}, \& \bibinfo{author}{McCall, B.~P.} (\bibinfo{year}{2002}). \newblock \bibinfo{title}{A temporal investigation of factors related to timely degree completion}. \newblock {\it \bibinfo{journal}{The Journal of Higher Education}\/}, {\it \bibinfo{volume}{73}\/}, \bibinfo{pages}{555--581}. %Type = Article \bibitem[{Dey \& Astin(1993)}]{Dey93} \bibinfo{author}{Dey, E.~L.}, \& \bibinfo{author}{Astin, A.~W.} (\bibinfo{year}{1993}). \newblock \bibinfo{title}{Statistical alternatives for studying college student retention: A comparative analysis of logit, probit, and linear regression}. \newblock {\it \bibinfo{journal}{Research in Higher Education}\/}, {\it \bibinfo{volume}{34}\/}, \bibinfo{pages}{569--581}. %Type = Article \bibitem[{Domingos \& Pazzani(1997)}]{domingos97optimality} \bibinfo{author}{Domingos, P.}, \& \bibinfo{author}{Pazzani, M.~J.} (\bibinfo{year}{1997}). \newblock \bibinfo{title}{On the optimality of the simple bayesian classifier under zero-one loss}. \newblock {\it \bibinfo{journal}{Machine Learning}\/}, {\it \bibinfo{volume}{29}\/}, \bibinfo{pages}{103--130}. %Type = Inproceedings \bibitem[{Druzdzel \& Glymour(1994)}]{Dru94} \bibinfo{author}{Druzdzel, M.~J.}, \& \bibinfo{author}{Glymour, C.} (\bibinfo{year}{1994}). \newblock \bibinfo{title}{Application of the {TETRAD II} program to the study of student retention in u.s. colleges}. \newblock In {\it \bibinfo{booktitle}{Working notes of the AAAI-94 Workshop on Knowledge Discovery in Databases (KDD-94)}\/} (pp. \bibinfo{pages}{419--430}). \newblock \bibinfo{address}{Seattle, WA}. %Type = Article \bibitem[{Ehrenberg \& Zhang(2005)}]{ehrenberg2005tenured} \bibinfo{author}{Ehrenberg, R.}, \& \bibinfo{author}{Zhang, L.} (\bibinfo{year}{2005}). \newblock \bibinfo{title}{{Do Tenured and Tenure-Track Faculty Matter?}} \newblock {\it \bibinfo{journal}{Journal of Human Resources}\/}, {\it \bibinfo{volume}{40}\/}, \bibinfo{pages}{647}. %Type = Inproceedings \bibitem[{Freund \& Mason(1999)}]{Freund99thealternating} \bibinfo{author}{Freund, Y.}, \& \bibinfo{author}{Mason, L.} (\bibinfo{year}{1999}). \newblock \bibinfo{title}{The alternating decision tree learning algorithm}. \newblock In {\it \bibinfo{booktitle}{In Machine Learning: Proceedings of the Sixteenth International Conference}\/} (pp. \bibinfo{pages}{124--133}). \newblock \bibinfo{publisher}{Morgan Kaufmann}. %Type = Article \bibitem[{Glynn et~al.(2003)Glynn, Sauer \& Miller}]{glynn2003} \bibinfo{author}{Glynn, J.}, \bibinfo{author}{Sauer, P.}, \& \bibinfo{author}{Miller, T.} (\bibinfo{year}{2003}). \newblock \bibinfo{title}{{Signaling student retention with prematriculation data}}. \newblock {\it \bibinfo{journal}{NASPA Journal}\/}, {\it \bibinfo{volume}{41}\/}, \bibinfo{pages}{41--67}. %Type = Inproceedings \bibitem[{Hall(2000)}]{Hall00correlation-basedfeature} \bibinfo{author}{Hall, M.} (\bibinfo{year}{2000}). \newblock \bibinfo{title}{{Correlation-based Feature Selection for Discrete and Numeric Class Machine Learning}}. \newblock In {\it \bibinfo{booktitle}{Proceedings of the Seventeenth International Conference on Machine Learning (ICML-2000): June 29-July 2, 2000, Stanford University}\/} (p. \bibinfo{pages}{359}). \newblock \bibinfo{organization}{Morgan Kaufmann}. %Type = Techreport \bibitem[{Heckerman(1996)}]{Heckerman96atutorial} \bibinfo{author}{Heckerman, D.} (\bibinfo{year}{1996}). \newblock {\it \bibinfo{title}{A Tutorial on Learning With Bayesian Networks}\/}. \newblock \bibinfo{type}{Technical Report} Learning in Graphical Models. %Type = Article \bibitem[{Herzog(2005)}]{Her2005} \bibinfo{author}{Herzog, S.} (\bibinfo{year}{2005}). \newblock \bibinfo{title}{Measuring determinants of student return vs. dropout/stopout vs. transfer: A first-to-second year analysis of new freshmen}. \newblock {\it \bibinfo{journal}{Research in Higher Education}\/}, {\it \bibinfo{volume}{46}\/}, \bibinfo{pages}{883--928}. %Type = Article \bibitem[{Herzog(2006)}]{Her2006} \bibinfo{author}{Herzog, S.} (\bibinfo{year}{2006}). \newblock \bibinfo{title}{Estimating student retention and degree-completion time: Decision trees and neural networks vis-à-vis regression}. \newblock {\it \bibinfo{journal}{New Directions for Institutional Research}\/}, {\it \bibinfo{volume}{131}\/}. %Type = Article \bibitem[{Holte(1993)}]{holte93} \bibinfo{author}{Holte, R.} (\bibinfo{year}{1993}). \newblock \bibinfo{title}{Very simple classification rules perform well on most commonly used datasets}. \newblock {\it \bibinfo{journal}{Machine Learning}\/}, {\it \bibinfo{volume}{11}\/}, \bibinfo{pages}{63}. %Type = Article \bibitem[{Jacoby(2006)}]{jacoby2006effects} \bibinfo{author}{Jacoby, D.} (\bibinfo{year}{2006}). \newblock \bibinfo{title}{{Effects of part-time faculty employment on community college graduation rates}}. \newblock {\it \bibinfo{journal}{Journal of Higher Education}\/}, {\it \bibinfo{volume}{77}\/}, \bibinfo{pages}{1081--1103}. %Type = Article \bibitem[{John(2000)}]{Joh2000} \bibinfo{author}{John, E.~P.} (\bibinfo{year}{2000}). \newblock \bibinfo{title}{The impact of student aid on recruitment and retention: What the research indicates}. \newblock {\it \bibinfo{journal}{New Directions for Student Services}\/}, (pp. \bibinfo{pages}{61--76}). %Type = Article \bibitem[{Lau(2003)}]{Lau2003} \bibinfo{author}{Lau, L.~K.} (\bibinfo{year}{2003}). \newblock \bibinfo{title}{Institutional factors affecting student retention}. \newblock {\it \bibinfo{journal}{Education}\/}, {\it \bibinfo{volume}{124}\/}, \bibinfo{pages}{126--137}. %Type = Article \bibitem[{Lotkowski et~al.(2004)Lotkowski, Robbins \& Noeth}]{Lot2004} \bibinfo{author}{Lotkowski, V.}, \bibinfo{author}{Robbins, S.}, \& \bibinfo{author}{Noeth, R.} (\bibinfo{year}{2004}). \newblock \bibinfo{title}{The role of academic and non-academic factors in improving college retention}. \newblock {\it \bibinfo{journal}{ACT Office of Policy Research}\/}, . %Type = Inproceedings \bibitem[{Massa \& Puliafito(1999)}]{Mas99} \bibinfo{author}{Massa, S.}, \& \bibinfo{author}{Puliafito, P.} (\bibinfo{year}{1999}). \newblock \bibinfo{title}{An application of data mining to the problem of the university students' dropout using markov chains}. \newblock In {\it \bibinfo{booktitle}{Principles of Data Mining and Knowledge Discovery. Third European Conference, PKDD'99}\/} (pp. \bibinfo{pages}{51--60}). \newblock \bibinfo{address}{Prague, Czech Republic}. %Type = Article \bibitem[{Menzies et~al.(2007)Menzies, Dekhtyar, Distefano \& Greenwald}]{me07e} \bibinfo{author}{Menzies, T.}, \bibinfo{author}{Dekhtyar, A.}, \bibinfo{author}{Distefano, J.}, \& \bibinfo{author}{Greenwald, J.} (\bibinfo{year}{2007}). \newblock \bibinfo{title}{Problems with precision}. \newblock {\it \bibinfo{journal}{IEEE Transactions on Software Engineering}\/}, . \newblock \bibinfo{note}{\url{http://menzies.us/pdf/07precision.pdf}}. %Type = Inproceedings \bibitem[{Menzies \& Hu(2007)}]{me07} \bibinfo{author}{Menzies, T.}, \& \bibinfo{author}{Hu, Y.} (\bibinfo{year}{2007}). \newblock \bibinfo{title}{Just enough learning (of association rules): The {TAR2} treatment learner}. \newblock In {\it \bibinfo{booktitle}{Artificial Intelligence Review}\/}. \newblock \bibinfo{note}{Available from \url{http://menzies.us/pdf/07tar2.pdf}}. %Type = Book \bibitem[{Mitchell(1997)}]{Mitchell97} \bibinfo{author}{Mitchell, T.~M.} (\bibinfo{year}{1997}). \newblock {\it \bibinfo{title}{Machine Learning}\/}. \newblock \bibinfo{address}{New York}: \bibinfo{publisher}{McGraw-Hill}. %Type = Book \bibitem[{Moore \& Notz(2006)}]{Moore06} \bibinfo{author}{Moore, D.}, \& \bibinfo{author}{Notz, W.} (\bibinfo{year}{2006}). \newblock {\it \bibinfo{title}{Statistics: concepts and controversies}\/}. \newblock \bibinfo{publisher}{WH Freeman \& Co}. %Type = Article \bibitem[{Murtaugh et~al.(1999)Murtaugh, Burns \& Schuster}]{Mur99} \bibinfo{author}{Murtaugh, P.~A.}, \bibinfo{author}{Burns, L.~D.}, \& \bibinfo{author}{Schuster, J.} (\bibinfo{year}{1999}). \newblock \bibinfo{title}{Predicting the retention of university students}. \newblock {\it \bibinfo{journal}{Research in Higher Education}\/}, {\it \bibinfo{volume}{40}\/}, \bibinfo{pages}{355--371}. %Type = Misc \bibitem[{NCPPHE(2007)}]{NCP2007} \bibinfo{author}{NCPPHE} (\bibinfo{year}{2007}). \newblock \bibinfo{title}{Retention rates - first-time college freshmen returning their second year ({ACT})}. %Type = Article \bibitem[{Pascarella \& Terenzini(1979)}]{Pas79} \bibinfo{author}{Pascarella, E.~T.}, \& \bibinfo{author}{Terenzini, P.~T.} (\bibinfo{year}{1979}). \newblock \bibinfo{title}{Interaction effects in spady and tinto's conceptual models of college attrition}. \newblock {\it \bibinfo{journal}{Sociology of Education}\/}, {\it \bibinfo{volume}{52}\/}, \bibinfo{pages}{197--210}. %Type = Article \bibitem[{Pascarella \& Terenzini(1980)}]{Pas80} \bibinfo{author}{Pascarella, E.~T.}, \& \bibinfo{author}{Terenzini, P.~T.} (\bibinfo{year}{1980}). \newblock \bibinfo{title}{Predicting freshman persistence and voluntary dropout decisions from a theoretical model}. \newblock {\it \bibinfo{journal}{The Journal of Higher Education}\/}, {\it \bibinfo{volume}{51}\/}, \bibinfo{pages}{60--75}. %Type = Phdthesis \bibitem[{Pittman(2008)}]{Pittman2008} \bibinfo{author}{Pittman, K.} (\bibinfo{year}{2008}). \newblock {\it \bibinfo{title}{Comparison of data mining techniques used to predict student retention}\/}. \newblock Ph.D. thesis Nova Southeastern University. %Type = Book \bibitem[{Quinlan(1986)}]{quinlanID3} \bibinfo{author}{Quinlan, J.~R.} (\bibinfo{year}{1986}). \newblock {\it \bibinfo{title}{Induction of decision trees}\/}. \newblock (\bibinfo{edition}{1st} ed.). %Type = Book \bibitem[{Quinlan(1993)}]{quinlanC4.5} \bibinfo{author}{Quinlan, J.~R.} (\bibinfo{year}{1993}). \newblock {\it \bibinfo{title}{C4.5: Programs for Machine Learning (Morgan Kaufmann Series in Machine Learning)}\/}. \newblock (\bibinfo{edition}{1st} ed.). \newblock \bibinfo{publisher}{Morgan Kaufmann}. %Type = Inproceedings \bibitem[{Rish()}]{NB-performance} \bibinfo{author}{Rish, I.} (). \newblock \bibinfo{title}{An empirical study of the naive bayes classifier}. \newblock In {\it \bibinfo{booktitle}{IJCAI-01 workshop on "Empirical Methods in AI"}\/}. \newblock \bibinfo{note}{\url{http://www.intellektik.informatik.tu-darmstadt.de/~tom/I% JCAI01/Rish.pdf}}. %Type = Article \bibitem[{Salazar et~al.(2004)Salazar, Gosalbez, Bosch, Miralles \& Vergara}]{Sal2004} \bibinfo{author}{Salazar, A.}, \bibinfo{author}{Gosalbez, J.}, \bibinfo{author}{Bosch, I.}, \bibinfo{author}{Miralles, R.}, \& \bibinfo{author}{Vergara, L.} (\bibinfo{year}{2004}). \newblock \bibinfo{title}{A case study of knowledge discovery on academic achievement, student desertion and student retention}. \newblock {\it \bibinfo{journal}{Information Technology: Research and Education, 2004. ITRE 2004. 2nd International Conference on}\/}, (pp. \bibinfo{pages}{150--154}). %Type = Inproceedings \bibitem[{Sanjeev \& Zytkow(1995)}]{San95} \bibinfo{author}{Sanjeev, A.}, \& \bibinfo{author}{Zytkow, J.} (\bibinfo{year}{1995}). \newblock \bibinfo{title}{Discovering enrolment knowledge in university databases}. \newblock In {\it \bibinfo{booktitle}{First International Conference on Knowledge Discovery and Data Mining}\/} (pp. \bibinfo{pages}{246--51}). \newblock \bibinfo{address}{Montreal, Que., Canada}. %Type = Inproceedings \bibitem[{Scalise et~al.(2000)Scalise, Besterfield-Sacre, Shuman \& Wolfe}]{Sca2000} \bibinfo{author}{Scalise, A.}, \bibinfo{author}{Besterfield-Sacre, M.}, \bibinfo{author}{Shuman, L.}, \& \bibinfo{author}{Wolfe, H.} (\bibinfo{year}{2000}). \newblock \bibinfo{title}{First term probation: models for identifying high risk students}. \newblock In {\it \bibinfo{booktitle}{30th Annual Frontiers in Education Conference}\/} (pp. \bibinfo{pages}{F1F/11--16 vol.1}). \newblock \bibinfo{address}{Kansas City, MO, USA}: \bibinfo{publisher}{Stripes Publishing}. %Type = Article \bibitem[{Spady(1970)}]{Spa70} \bibinfo{author}{Spady, W.~G.} (\bibinfo{year}{1970}). \newblock \bibinfo{title}{Dropouts from higher education: An interdisciplinary review and synthesis}. \newblock {\it \bibinfo{journal}{Interchange}\/}, {\it \bibinfo{volume}{1}\/}, \bibinfo{pages}{64--85}. %Type = Article \bibitem[{Spady(1971)}]{Spa71} \bibinfo{author}{Spady, W.~G.} (\bibinfo{year}{1971}). \newblock \bibinfo{title}{Dropouts from higher education: Toward an empirical model}. \newblock {\it \bibinfo{journal}{Interchange}\/}, {\it \bibinfo{volume}{2}\/}, \bibinfo{pages}{38--62}. %Type = Article \bibitem[{Stage(1989)}]{Sta89} \bibinfo{author}{Stage, F.} (\bibinfo{year}{1989}). \newblock \bibinfo{title}{{Motivation, Academic and Social Integration, and the Early Dropout}}. \newblock {\it \bibinfo{journal}{American Educational Research Journal}\/}, {\it \bibinfo{volume}{26}\/}, \bibinfo{pages}{385--402}. %Type = Misc \bibitem[{Stewart \& Levin(2001)}]{Ste2001} \bibinfo{author}{Stewart, D.~L.}, \& \bibinfo{author}{Levin, B.~H.} (\bibinfo{year}{2001}). \newblock \bibinfo{title}{A model to marry recruitment and retention: A case study of prototype development in the new administration of justice program at blue ridge community college}. %Type = Article \bibitem[{Sujitparapitaya(2006)}]{Suj2006} \bibinfo{author}{Sujitparapitaya, S.} (\bibinfo{year}{2006}). \newblock \bibinfo{title}{Considering student mobility in retention outcomes}. \newblock {\it \bibinfo{journal}{New Directions for Institutional Research}\/}, {\it \bibinfo{volume}{2006}\/}. %Type = Inproceedings \bibitem[{Superby et~al.(2006)Superby, Vandamme \& Meskens}]{Sup2006} \bibinfo{author}{Superby, J.~F.}, \bibinfo{author}{Vandamme, J.~P.}, \& \bibinfo{author}{Meskens, N.} (\bibinfo{year}{2006}). \newblock \bibinfo{title}{Determination of factors influencing the achievement of the first-year university students using data mining methods}. \newblock In {\it \bibinfo{booktitle}{8th International Conference on Intelligent Tutoring Systems (ITS 2006)}\/} (pp. \bibinfo{pages}{37--44}). \newblock \bibinfo{address}{Jhongli, Taiwan}. %Type = Article \bibitem[{Terenzini \& Pascarella(1980)}]{Ter80} \bibinfo{author}{Terenzini, P.~T.}, \& \bibinfo{author}{Pascarella, E.~T.} (\bibinfo{year}{1980}). \newblock \bibinfo{title}{Toward the validation of tinto's model of college student attrition: A review of recent studies}. \newblock {\it \bibinfo{journal}{Research in Higher Education}\/}, {\it \bibinfo{volume}{12}\/}, \bibinfo{pages}{271--282}. %Type = Misc \bibitem[{Tillman \& Burns(2000)}]{Till} \bibinfo{author}{Tillman, C.}, \& \bibinfo{author}{Burns, P.} (\bibinfo{year}{2000}). \newblock \bibinfo{title}{{Presentation on First Year Experience}}. \newblock \bibinfo{note}{\url{http://www.valdosta.edu/~cgtillma/powerpoint.ppt}}. %Type = Article \bibitem[{Tinto(1975)}]{Tin75} \bibinfo{author}{Tinto, V.} (\bibinfo{year}{1975}). \newblock \bibinfo{title}{Dropout from higher education: A theoretical synthesis of recent research}. \newblock {\it \bibinfo{journal}{Review of Educational Research}\/}, {\it \bibinfo{volume}{45}\/}, \bibinfo{pages}{89--125}. %Type = Article \bibitem[{Tinto(1982)}]{Tin82} \bibinfo{author}{Tinto, V.} (\bibinfo{year}{1982}). \newblock \bibinfo{title}{{Limits of Theory and Practice in Student Attrition}}. \newblock {\it \bibinfo{journal}{The Journal of Higher Education}\/}, {\it \bibinfo{volume}{53}\/}, \bibinfo{pages}{687--700}. %Type = Article \bibitem[{Tinto(1988)}]{Tin88} \bibinfo{author}{Tinto, V.} (\bibinfo{year}{1988}). \newblock \bibinfo{title}{Stages of student departure: Reflections on the longitudinal character of student leaving}. \newblock {\it \bibinfo{journal}{Journal of Higher Education}\/}, {\it \bibinfo{volume}{59}\/}, \bibinfo{pages}{438--455}. %Type = Article \bibitem[{Vandamme(2007)}]{Van07} \bibinfo{author}{Vandamme, J.} (\bibinfo{year}{2007}). \newblock \bibinfo{title}{{Predicting Academic Performance by Data Mining Methods}}. \newblock {\it \bibinfo{journal}{Education Economics}\/}, {\it \bibinfo{volume}{15}\/}, \bibinfo{pages}{405--419}. %Type = Misc \bibitem[{Veitch(2004)}]{Vei2004} \bibinfo{author}{Veitch, W.~R.} (\bibinfo{year}{2004}). \newblock \bibinfo{title}{Identifying characteristics of high school dropouts: Data mining with a decision tree model}. %Type = Misc \bibitem[{Waugh et~al.(1994)Waugh, Micceri \& Takalkar}]{Wau94} \bibinfo{author}{Waugh, G.}, \bibinfo{author}{Micceri, T.}, \& \bibinfo{author}{Takalkar, P.} (\bibinfo{year}{1994}). \newblock \bibinfo{title}{Using ethnicity, {SAT/ACT} scores, and high school {GPA} to predict retention and graduation rates}. %Type = Book \bibitem[{Witten \& Frank(2005)}]{Wit2005} \bibinfo{author}{Witten, I.}, \& \bibinfo{author}{Frank, E.} (\bibinfo{year}{2005}). \newblock {\it \bibinfo{title}{Data Mining: Practical Machine Learning Tools and Techniques}\/}. \newblock (\bibinfo{edition}{2nd} ed.). \newblock \bibinfo{address}{San Francisco}: \bibinfo{publisher}{Morgan Kaufmann Publishers}. %Type = Misc \bibitem[{Yu et~al.(2007)Yu, DiGangi, Jannasch-Pennell, Lo \& Kaprolet}]{YuD2007} \bibinfo{author}{Yu, C.~H.}, \bibinfo{author}{DiGangi, S.}, \bibinfo{author}{Jannasch-Pennell, A.}, \bibinfo{author}{Lo, W.}, \& \bibinfo{author}{Kaprolet, C.} (\bibinfo{year}{2007}). \newblock \bibinfo{title}{A data-mining approach to differentiate predictors of retention between online and traditional students}. %Type = Article \bibitem[{Zhang \& Zhang(2007)}]{zhang07} \bibinfo{author}{Zhang, H.}, \& \bibinfo{author}{Zhang, X.} (\bibinfo{year}{2007}). \newblock \bibinfo{title}{Comments on 'data mining static code attributes to learn defect predictors'}. \newblock {\it \bibinfo{journal}{IEEE Transactions on Software Engineering}\/}, . %Type = Article \bibitem[{{\.Z}ytkow \& Zembowicz(1993)}]{Zyt1993} \bibinfo{author}{{\.Z}ytkow, J.}, \& \bibinfo{author}{Zembowicz, R.} (\bibinfo{year}{1993}). \newblock \bibinfo{title}{Database exploration in search of regularities}. \newblock {\it \bibinfo{journal}{Journal of Intelligent Information Systems}\/}, {\it \bibinfo{volume}{2}\/}, \bibinfo{pages}{39--81}. \end{thebibliography} \end{document}