\section{Drastic vs. Conservative Control (Hoh In strategies)}
Earlier we presented a paper by Boehm that suggested using strategies  in order to impose control over a project and resolve conflicts in project requirements. These strategies were presented earlier, and are meant to override internal project parameters and policies. For this reason we have placed these strategies under the guise of drastic change. In the software world, these drastic solutions to conflicts within projects are suggested that would theoretically modify a project in the intended  manner. However these drastic solutions can have unintended effects , where a strategy like reduce functionality might deliver the effect of reduce cost and time of development, but also put in the risk of under delivering. This would not be helpful for the project as that might cause it to be poorly received. 

Conservative project changes are changes with in the project that abide by preset project parameters, which is the way that star operates by default. We wish to present the following argument: Before performing {\em drastic changes} to a project,  it is  worthwhile to
thoroughly explore the available options within the current structure of a
project.  Through applying policies that avoid drastic change, we wish to see whether we are able to achieve the same or better effects compared to drastic changes, while at the same time attempting to avoid the negative side effects that drastic changes inherently have.

\input{figures/misc/how}

The above figure presents the various strategies presented by Boehm et al., and there implementation within STAR. Note that the setting on the right hand side overrides any project settings that conflict with them. All the numbers indicate ratings, with KSLOC being the only exception, where the number is a ratio that is multiplied by the range that the project has in for the KSLOC attribute.

\input{figures/hohin/winloss-hohin}

As we did in the previous section, this section will also only concentrate on analysing the results for three of the four software engineering models used within STAR. The experiments for each scenario was run ten times, after which the data generated at the policy points was collected and passed through the Mann-Whitney unit test~\cite{mann47}, where each strategy is ranked. The total number of runs of STAR in this case study is 
\begin{center}
$strategies*projects*10$
\end{center}
 This calculates to a total of 400 runs, since we used the NASA projects. Note that the above figures are quartile charts that indicate the variations in the runs. Only the second and third quartile are shown in order to reduce the clutter of the graphs that would've been caused by outliers. ``\textbf{STAR}'' indicates conservative change, with non of the strategies being used. Also, the number on the left of the quartile indicate the rank according to Mann-Whitney. Threats weren't included in this study due to results that had it zeroed out accross the board, with occational outliers which produced messy, inconclusive results, a pattern seen several throughout this thesis.
 
In terms of the results for effort, clearly avoiding any drastic change within the project  is the best choice, ranking first in all the tests. And while it did tie with other strategies for the different projects, conservative change maintained its rank across the multiple projects, irrespective of how constrained a project is. Another clear result is that improving programmer capability (pcap) is definitely not the way to go to maintain costs, ranking last for every project. Other notable observations include the reducing functionality does well until you hit the most constrained projects and that improving the tools used and the platform does best in the most constrained project. Note here however that, despite of the ranking, most of the alternative strategies are within ballpark figures of each other, except for the lowest ranking strategies, making those the most definitive result.

Moving on to development time, there seems to be a similar picture compared to the results under effort. Improving \textit{pcap} is still the worst thing you can do, and avoiding drastic change is still a favorable policy to go by. At worst, conservative change manages to be second for the most constrained project. On the other hand reducing functionality and relaxing the schedule both do very well with the lightly constrained projects (flight and ground), and improving the tools and platform used does best in the most constrained project. One very interesting definitive result comes from reduce quality in OSP2, where this strategy wins out by a large margin, while it scored mid-pack in all the other projects. This could be interpreted as saying the following: earlier in the project, reducing the quality of the product doesn't help with development time; however, as the development of the product nears the end stages, if you need to rush the project out, reducing the quality (which happens to reduce reliability and documentation) will help in getting the product out the door.  A word of caution here however: this short term payoff only comes at a cost in the medium term. As can be seen in the OSP2 results for effort in \fig{hohineffort} and defects in \fig{hohindefects}, the side effect of rushing a product out is larger effort and much larger defect numbers.

Finally, when it comes to defects, we see a consistent overall result that reducing quality will in fact increase defects. This is demonstrated by having \textit{reduce quality} rank last in all the projects, with an especially large margin in the more constrained project (OSP, OSP2). Here as well though, conservative change does well. It scores lowest with ground; however, looking at the quartile charts for ground, the difference between STAR and the higher ranked drastic change strategies isn't significant enough to declare a clear loss.

To conclude this section, we have and demonstrated that, overall, conservative change and policies that abide by project limits are a better, more consistent choice for project managers. Even though drastic change does seem to do better in some cases, the difference is either not significant enough or the strategy itself has serious side effects on other aspects of the project. 

\section{Better, Faster, Cheaper}
Having presented the history of BFC at NASA, our purpose behind this section is to attempt to investigate the viability of achieving a project that has high reliability and quality, was cheap to develop and was quick to get to market. The general consensus before the Goldin era at NASA was that you had to have a compromise somewhere. Hence the common phrase "Better, Faster, Cheaper: pick any two". 

To apply study this, we needed to apply the different scenarios of BFC within STAR. These include BFC, FC, BF and BC, where the latter three options are the ``pick any two'' variants. For this study, BFC was interpreted as an indication of what models the weighting is on in~\eq{energy}. More specifically, it was interpreted as the following:
\bi
\item
BFC: maintain default weights in the fitness function: $\alpha$ = 1, $\beta$ = 1, $\gamma$ = 1, $\delta$ = 1 and $relydefect$ = 1.8.
\item
FC: Apply higher weight on the effort and development time aspects in the fitness function, and eliminate all the others: $\alpha$ = 2, $\beta$ = 0, $\gamma$ = 0, $\delta$ = 2 and $relydefect$ = 0.
\item
BF: Apply higher weight on the defects and development time aspects in the fitness function, and eliminate all the others: $\alpha$ = 0, $\beta$ = 2, $\gamma$ = 0, $\delta$ = 2 and $relydefect$ = 1.8.
\item
BC: Apply higher weight on the defects and effort aspects in the fitness function, and eliminate all the others: $\alpha$ = 2, $\beta$ = 2, $\gamma$ = 0, $\delta$ = 0 and $relydefect$ = 1.8.
\ei

\input{figures/bfc/winloss-bfc}

For each of the NASA projects presented in \tion{nasaprojpol}, STAR was run ten times for each of the variant weighting schemes, all while employing default analysis. The results from these runs were post processed to produced the quartile charts presnted in \fig{bfceffort}, \fig{bfctime}, and \fig{bfcdefects}. Note that these charts only show the middle two quartiles in an effort to keep the charts legible, and in order to avoid reporting outliers. These quartiles have also been normalized to further emphasize the differences between the different schemes. The rank field in those charts indicates the rank given to the particular weighting scheme when statistically compared to the other schemes using the Mann-Whitney unit test~\cite{mann47}. Threats weren't included in this study due to results that had it zeroed out accross the board, with occational outliers which produced messy, inconclusive results, a pattern seen several throughout this thesis.

Looking at \fig{bfceffort}, the figure describing effort results, we observe that BFC seems to perform mid-pack compared to the rest of the weighting schemes, while FC ranks first in all but one project (OSP2). For OSP2, BC ranks first among the weightings, with FC ranking last. Note that in all the other projects, BC actually ranked last. However, those rankings can't be considered a clear loss for BC since the range of effort estimates produced by BC overlaps fairly well with the estimates from the other weighting schemes. FC's loss in OSP2 however is a more clear loss.

Moving on to \fig{bfctime}, the figure describing development time results, we can again see how BFC performs mid-pack compared to the other schemes. This tiem aroung, FC wins accross the board, with BF trailing close behind, and BC rounding out the pack. The more decisive wins for FC take place with the more constrained projects, OSP and OSP2, as with the more decisive losses of BC.

Finally, in \fig{bfcdefects}, the figure describing defects results, we observe that BFC does very well with OSP and OSP2, perfoming on par with BC and BF for these projects. On the other hand, FC does very poorly for these projects, clearly driving defects up due to a lack of emphasis on defects. The performance in the other two projects is closer, however note that BFC does slightly worse than everything else with \textit{ground}.

From the above observations, we present the following analysis relating to BFC:
\be
\item
BFC is a ``jack of all trades, master of none''. It works well in most instances, and provides a compromise solution in balancig out model estimates. This supports the view that you can't fully acheive BFC with out some compromises, and that you're better off ``picking any two'' aspects to concentrated if there is a focused end result is needed.
\item
FC speeds  projects, with having the side effect of drastically increasing defects for more defined and constrained projects. This seems to suggest against rushing well defined and constrained projects out the door, and concentrating on on keeping some ``better'' aspect in consideration to avoid the defect spike, especially for mission critical projects.
\item
Finally that BC, while driving down defects, also drives up development time, especially for more constrained projects. This also applies for effort for all projects other than the most constrained (OSP2)
\ee

So as a final conclusion to this section, this study seems to support ``better, faster, cheaper: pick any two'', providing an empirical study that opposes the BFC policies applied throughout NASA in the 1990's to the early 2000's.