
\subsection{Improving Software Projects with {\W}}

%Before we can offer evidence of $\Delta$-resilience in our work with {\W}, we present the following case study demonstrating the {\W} algorithm. As an instance-based planner, {\W} operates entirely on historical cases, making no assumptions of underlying distributions or domain-specific models. Summarized, {\W} suggests changes that occur more frequently in desirable cases and less often in undesirable cases.

{\W} requires a set of historical cases as well as a defined $context$ of controllable feature ranges within those cases. Historical cases are expressed via a set of $P$ project descriptors such as analyst capability, function points, schedule constraints, etc. Each of these cases must also be labed with some known goal metric, such as development effort in man-months. The $context$ consists of a subset of the project descriptors that describe the project at hand, as well as which descriptors are controllable. {\W} uses the entire $context$ to determine the relevant neighborhood, then restricts recommendations to the controllable descriptors only.

For example, a company may track prior software projects via the COCOMO descriptors for analyst capability (acap), schedule requirements (sced), lines of code (kloc), and so on. When planning with {\W}, the project manager could define a new project using all available descriptors. However, business decisions (such as a hiring freeze on analysts) would constrain the controllable descriptors to only those deemed implementable. From these controllables, {\W} can begin reasoning.


{\W} assumes that a manager can offer us (a) a description of the  $context\subseteq P$ that interests them and (b) a list of $control$able options which they can change ($control \subseteq context$). For example, consider the following context for a Finnish banking software project. For simplicity's sake, each descriptor is rated either high, medium, or low:
\[\small
\begin{array}{r@{~}c@{~}l}
\multicolumn{3}{l}{context_1 =}\\ 
\mbox{~~~~~~}&  & ?hw \in \left\{lo, md \right\}\wedge 
 ?at\in \left\{md, hi\right\} \wedge
     ?FP\in \left\{lo, md \right\} \wedge \\
& & ?co\in \left\{lo, md\right\} \wedge
    ?prod\in \left\{md, hi \right\} \wedge
 size\in \left\{md, hi\right\}\wedge\\
\end{array}
\]
Here,  ``?'' are the $control$ labels; the manager is only concerned with projects that have a low or average number of function points (FP).

{\W} finds a project treatment $R_x$ by studying the project similar to the context in the case repository. Formally, {\W}2 explores the $neighorhood$ of the $context$, looking for ways to select for the ``best'' cases using some utility measure. In this case, minimizing software effort requirements.
In {\W}, this is a six step procedure:
\be
\item Divide cases randomly into $train:test$ in the ratio $N_1:N_2$.
\item Use $context$ to find $K$ nearest-neighbors within $train$.
\item Using the utility measure, the neighborhood into (a)~the $best$ cases that should be emulated, and (b) the remaining cases to avoid (which we call $rest$).
\item Rank all differences between  (a) and (b) according to how strongly they select for the $best$ cases.
\item Use the $train$ set again, experiment with treatments $R_x$ built from the  top ranked items found in $Step4$.
Return the treatment that  selects for the cases in the $train$ set with highest median $value$.
\item Test the treatment from $Step6$ using relevant cases from the $test$ set; i.e. find all rows in the neighborhood of the $context$ in $test$ set; then find the subset of those rows that match  the treatment.
\ee
The current implementation of $W$ using $N_1=2$, $N_1=1$, $K=20$.
%\section{Data}
%\input{data}
%As stated prior, {\W} operates on the controllable space of project ranges from historical cases. Because {\W} operates on discrete, actionable changes to software projects, continuous features must be discretized. This requirement initially lead us to the concept of $\Delta$-resiliency. Our initial concern was that upon discretization, performance would drop as data was inserted into coarse bins.
%
%The precision expressed in these datasets demonstrates our motivations in discussing $\Delta$-resiliency. For example, the Finnish dataset represents a the $prod$ feature as a floating-point number accurate to \emph{six} decimal places. If equivalent effort reduction could be achieved without such precision, then a business need not expend resources in obtaining that precision.
%
% \fig{data} describes the number of features (Cols) and instances (Rows) for each dataset. All of these datasets are freely available from the PROMISE repository (\footnote{PROMISE: http://promisedata.org/data}). Also, while the datasets used in this paper describe projects scored by development effort, there is no formal requirement that {\W} be presented with effort data. We offer a case study only in effort reduction to better allow comparision between results.
%
The following case study demonstrates one run of {\W} using the $Finnish$\footnote{PROMISE: http://promisedata.org/data} dataset. After setting aside $2/3$rds of the data for training in $Step1$, \fig{train-filter} shows the most neighborhood generated in $Step2$. Each instance is ranked by how many features fall within $context$, labed as ``overlap'' in \fig{train-filter}. Cells in gray represent ranges that fall outside $context$. The top 20 rows with the highest overlap are selected.

\input{figs/train-filter.tex}

Next, $Step3$ sorts the cases by utility, with $K_1=5$ rows placed into the $best$ set and the remaining $K_2=15$ rows placed into the $rest$ set (\fig{train-contrast}). The \emph{contrast set} between these two sets is shown in \fig{train-bestrest}. Frequency counts are computed for each discrete feature value, with each potential treatment $R_x$ ranked as follows:
\begin{equation}\label{eq:ps}\small
rank(E) * support(E) = \frac{ratio(E|best)^2}{ratio(E|best) + ratio(E|rest)}
\end{equation}

\input{figs/train-contrast.tex}
\input{figs/train-bestrest.tex}

Note that all treatments are ranked in \fig{train-bestrest}, even those that never occur in the $best$ set. To decide which treatments should be included as the final recommendation, the best to worst ($R_1$, $R_2$, ...) individual treatments are applied to the $train$ set. Each time a treatment is added, the $train$ set is constrained to rows that only match that treatment. The median utility of the constrainted set is compared to the original $train$ set.
$Step5$  explores $R_x$ upwards from $x=1$ while:
\bi
\item
The median $value$ of the rows selected by $R_{x+1}$ is  greater than that of $R_x$.
\item The number of selected rows $|R_x \wedge neighborhood| \ge 3$;
In the case of \fig{testing-constrained}, $Step5$ returns $R_1$ ($FP=lo$). 
\ei
\fig{testing-constrained} shows 
the cases from this neighborhood 
that satisfy $R_1 : FP=lo$.

\input{figs/test-constrain.tex}

\fig{testing-constrained} applies $R_1$  on the test data.  Its impact is reported as the median effort value of the cases. In the case of $FP=lo$, the constrained testing set gives a median effort of 6.67, a 31\% reduction from the original estimate of 8.8.

\section{results}

We ran {\W} across a series of discretized effort datasets defined in \fig{data}. For bin partitioning we used equal frequency discretization with the number of bins ranging from 2 to 10. All datasets except for the COCOMO-based NASA93 datasets were originally numeric.

As stated earlier, given $B'$ bins, we compute $\Delta$-resiliency by the smallest $B'$ such that optimizations from $B'-1$ bins are \emph{no worse} than $B'$ and optimizations from $B'+1$ bins are \emph{no better} than $B'$. At this point $\Delta$ defines the optimizer's resilience for up to $100 / B' / 4\%$ noise levels.

\fig{discrete-results} show the results of this experiement. Note that the leftmost column references the Mann-Whitney U ranked test. Bin sizes of equal rank are defined to be from the same distribution, asserting that performance is statistically \emph{no better} and \emph{no worse}. Consider the Finnish dataset. 4 bin discretization performs as well as 3 bin discretization, but no better than 5 bin discretization. From this we say that $\Delta(finnish|W) = 4$, asserting noise resistance for up to $100 / 4 / 4$ = 6.25\%.

For other datasets, the smallest $\Delta$-resilient bin size varies between 3 and 4 bins. This may come as a surprise, as Kemerer, Miyazaki, Telecom, China, and Finnish are all continuous datasets with some values carried out to \emph{six} decimal places. Yet, our learner performs just as well when reduced to simple ``low, nominal, high'' labelling of features.

Overall, our results demonstrate little need for fine granularity in data collection. It's quite possible to reduce data into course chunks, thus making data collection easier.

\input{figs/effort-results}

\input{figs/results-summary}

\includegraphics[width=3in]{figs/finnish-delta.pdf}
\includegraphics[width=3in]{figs/china-delta.pdf}
\includegraphics[width=3in]{figs/kem-delta.pdf}
