\section{JPL Studies}\label{sec:jpl}
%\input{tables/jpl-braille}
The purpose of this study is to observe how policy recommendations of STAR would change for ground and flight projects at JPL over time. Through correspondence with Jarius Hihn from JPL we were able to obtain past and current project specifications of flight and ground software projects according to the COCOMO-II model. These specifications are indicated in \fig{flight9000} through to \fig{ground7080}. Note that here the data ranges that are presented for the different JPL projects are based on actual project ranges from products that are merged together. However this is dependent on the on the amount of sample projects available, and instances like ground 2000 in \fig{ground9000} are due to only having a single sample project for that type and time period. Aside from this exception, the general trend for these project templates seems to be that older projects are more restricted than newer ones, having more set values for attributes, as well as tighter ranges for some of these attributes. This is a case of ``20-20 hindsight'', where we know more about the more mature past projects than the more recent projects.

\input{figures/JPLcases}
\input{tables/jpl-braille-collapsed}

%Note that these project ranges are based on the combination data of several projects of the same type (i.e. flight or ground) from the same time period, where in some cases there weren't much projects available to produce sufficient ranges. This was the case for JPL ground software circa 2000. 
In addition to the above project model descriptions, we were also given modified policy specifications that are catered to these flight and ground systems. These are indicated in \fig{strattact}. Note that the only difference in the specifications of the JPL flight and ground are in the tactical policy, where ground tactical adds \textit{time} and \textit{stor} in the search space.

\fig{jplp1} and \fig{jplp2} present the results of the STAR analysis in the form of braille charts. for the columns in these charts, ``s'' designates use of the strategic policy, while ``t'' designates use of the tactical policy. In addition, for the projects, ``f'' and ``g'' indicate flight and ground respectively, with the numbers following indicating which decade the particular project template belongs to. Note that any project recommendations produced by STAR are rounded up.

There are several results here that can be observed. One of these is a result that confirms our findings in \chap{ase} concerning the  importance of advanced software engineering tools. Here, the recommendation to use $aa=6$ and/or $ett=6$ appears $\dfrac{13}{16}$ times, noting how only the highest values available for these tools are recommended. In addition to this, we can also see how \textit{tool} is recommended across the board at its highest levels available to that particular project template. This results implies that, no matter the time period or the development tools available, project managers should seek to apply these tools at there highest levels available for that project.

Another result that can be observed is in the reductions that the recommendations produced by STAR generate, and how that relates to how constrained projects are. Notice how the most constrained projects (ground 2000, flight 1970 and 1980) have the least reductions. On the other hand, the most unconstrained project (flight 1990) exhibits that largest reductions compared to the initial estimates of that project template. This is a result that is also seen in \chap{ase}, as well as presented in a workshop paper by Menzies et. al~\cite{me07g}, where it is observed that ``if you fix everything, you have nothing else to fix''.

In the following section, we will present a study similar to this one, however applied to future projects in order to study possible future trends in software engineering. Also, the project templates themselves are meant to be more descriptive of software engineering practices at large and how leading experts see that field progressing in the medium term.

\section{Futures}\label{sec:future}

\subsection{Data Collection}
In this section we will describe how data was collected in order to provide some insight on future trends in the software industry. That will be followed with presenting the results of this data collection and how those were used in STAR.


\begin{figure}[!t]
\begin{center}
\begin{tabular}{cc}
\includegraphics[scale=0.45]{misc/workshop1.png}&\includegraphics[scale=0.45]{misc/workshop2.png}
\end{tabular}
\caption{Pictures of the panel at ICSE 2008.}\label{fig:pan}
\end{center}
\end{figure}



\begin{figure}
\scriptsize
\begin{center}
\begin{tabular}{l|c|l|c}
feature&setting&feature&setting\\\hline
aa&3&tool&3\\
acap&3&time&3\\
apex&4.5&team&3.5\\
cplx&3&stor&3\\
data&4&site&4.5\\
docu&3&sced&3\\
ett&2&ruse&4\\
flex&3&resl&3\\
ltex&4.5&rely&2.5\\
pcap&3&pvol&2\\
pcon&3&prec&4.5\\
peer&2.5&pmat&2.5\\
plex&4.5\\
\end{tabular}
\end{center}
\caption{The baseline settings representing present day software engineering}\label{fig:base1}
\end{figure}



\begin{figure}
\scriptsize
\begin{center}
\begin{tabular}{lrr|lr}
       &\multicolumn{1}{c}{ranges}      &\multicolumn{3}{c}{values}\\\hline
feature&low&high&feature&setting\\\hline
aa&3&6&data&4\\
acap&3&5&docu&3\\
apex&1&4.5&ltex&4.5\\
cplx&3&6&pcap&3\\
ett&2&6&plex&4.5\\
flex&3&5&team&3.5\\
pcon&1&3&&\\
peer&2.5&6&&\\
pmat&2.5&5&&\\
pvol&2&5&&\\
rely&2.5&5&&\\
resl&3&5&&\\
ruse&2&6&&\\
sced&1&3&&\\
site&1&4.5&&\\
tool&3&5&&\\
\end{tabular}
\end{center}
\caption{Common settings for all future trend variants}\label{fig:cases1}
\end{figure}

\normalsize

The panel meeting was conducted during ICSE 2008 on May 13. \fig{pan}
 are photos of this panel.
It was composed of seven seasoned experts in the field of software engineering:
\bi
\item
Barry Boehm
\item
Vic Basili
\item
Ray Madachy
\item
Thomas Ostrand
\item
Deiter Rombach
\item
Rick Selby
\item
Elaine Weyeuker
\ei

These experts formed the part of the panel from whom insights concerning the future trends in the software industry was derived. They discussed projections concerning all the COCOMO II/COQUALMO factors. Note that the time frame for these projections is within a decade. Other assumptions concerning the future projects include the following:
\bi
\item
The projects/applications are {\em NASA-like} applications.
\item
They are {\em mission critical real-time} applications.
\item
They are {\em contractor built}.
\item
They have a {\em long life time} (between 5 and 10 years).
\ei


Three facilitators were present as well:
\bi
\item
Bojan Cukic
\item
Tim Menzies
\item
David Raffo
\ei

The role of the facilitators was to guide the discussion on the model attribute projections in an attempt to keep the discussions focused. Gregory Gay acted as scribe during this session in order to transcribe notes from the meeting.

During discussions, consensus was rarely reached concerning projections about how the software environment might change relative to the models. For this reason, after brief discussions about every attribute, a vote was taken concerning the future behavior of the attribute. These behaviors include the following:
\bi
\item
\textbf{Bump}: The attribute would increase in value at first, and then decrease later on. PREC was voted to have such behavior.
\item
\textbf{Rick}: This is the opposite of bump, where attribute would decrease in value at first, and then increase later on.
\item
\textbf{Up}: The attribute would increase in value. ACAP was voted to have such behavior.
\item
\textbf{Down}: The attribute would decrease in value. APEX was voted to have such behavior.
\ei

Some chose to abstain from some of the votes as well. \fig{votes1} and \fig{votes2} 
contain the summary of the results of these discussions, as well as the votes.

Note that in order to have future projections, we had to set a baseline that represents the current trends in software engineering. This had been set prior to the panel session with input from one of the expert panelists, Barry Boehm, and is presented in \fig{base1}
. From this baseline, we branched 8 main projections. the reason being that, as we see in \fig{votes1} and \fig{votes2}
, only 3 attributes (PREC, STOR and TIME) were voted as not having a single direction of variation, namely bumps. \fig{cases1}
 shows all the other attributes that were voted to have a single direction of change, and there corresponding ranges and values.

\subsection{Policies}

When referring to \fig{exlarge}, \fig{exmedium} and \fig{exsmall}
, we notice that the policies that are chosen by STAR are very similar across project sizes and projections. This would seem to suggest that, even though we have 8 separate base projections, the overall projection is stable and determinant enough to actually draw stable conclusions from that would apply across the board. In other words, for the purposes of STAR, the experts panel from whom the projections originate don't necessarily need to have consensus for us to draw stable future predictions about policies that need to be applied in the future.

One interesting effect that we see here relates to ASE tools. On the one hand these tools are recommended for the majority of our case studies. This applies most to those cases with full or technical analysis, and to a lesser extent to those with strategic analysis. Note however that in all cases, whenever these tools are applied, they are applied at very high or extremely high levels 
(between 5 and 6), which emphasizes the importance of advanced ASE tools for future projects. These tools however have the most utility whenever the project is in the development phases. This explains our result since statistical analysis is more related to the planning stages of a project, which tactical analysis is more related to the development stage of a project. Also notice that for the same analysis methods the emphasize the use of advanced ASE tools, the TOOL is recommended at its highest value, which means having it integrated in to the life cycle according to \fig{cparems}. So there seems to be a correlation between having the need for ASE tools, and requiring having these tools integrated into the software life cycle.

\input{tables/futures-braille-2}

\subsection{Reductions}
The lower parts of \fig{exlarge}, \fig{exmedium} and \fig{exsmall} 
refer to the size of the final model prediction values compared to the initial ones, i.e. the ones before applying any kind of policies on the case study in question. So smaller is better. We notice four main effects in these tables.

The first effect is that the reductions are very similar across case studies of the same kind of analysis. This doesn't apply in all cases however. The exceptions included such cases where ASE tools aren't applied at all. These tend to have much lower reductions in defects compared to other cases with these tools were being applied. However, the overall effect seems to be that overall the reductions are similar for the same analysis types regardless of the projection or project size. This is mainly due to the similarity in the policies that are followed, which is mentioned in the previous section.

Secondly, effort reductions seem to be larger for strategic compared to tactical analysis. This would suggest that strategic long term planning is more effective at reducing cost for future projects. Note that our strategic analysis with STAR didn't decisively recommend the use of advanced ASE tools. So there seems to be an argument that ASE tools are necessary for cost reduction. However this isn't the whole picture.

We also observe that defect reductions are largest with tactical analysis. This is due to the affinity of this kind of analysis to recommend ASE tools be used. ASE tools are mainly defect reduction tools, so its only natural that the analysis that most recommends these tools achieves higher defect reductions. The fact that its tactical analysis suggests that ASE tools are best used during the development stages of a software product rather than during the planning stages.

The last effect that we observe is that full analysis achieves the best reductions by a large margin. This applies for all the models and all the case studies, suggesting that if we could control everything in a project, we should. This unfortunately isn't really possible nor feasible. However this can be interpreted as using both proper strategic and tactical planning when appropriate. So, use strategic planning on a future project, apply the policies, then transition to tactical planning and apply the associated policies. By doing this, a software project could achieve a similar effect to controlling all, or most, of the variables in a software project achieving high reductions in effort, defects, and months (schedule). Note that this type of analysis recommends used advanced ASE tools most of the time, providing a strong business case for them.

Note that the threat model doesn't seem to have much importance in our case studies in that model never seem to register any tangible values in any of the experiments. We have seen this before, where for less constrained case studies threat doesn't really come into play (\tion{aseexp}).

\begin{center}
\input{tables/icseMay13}

\input{tables/futures-summary}
\end{center}
