\begin{figure}
\begin{center} \includegraphics[width=0.5\textwidth]{plots/rank-changes-moe.pdf} \end{center}
\caption{Rank changes of solo and multi-methods. Region $1 \leq X \leq 9 $ contains 9 out of 12 multi-methods.
See that \textbf{Top13/Mean} at $X=1$ has a ${\delta}r$ of $1$, i.e. it outperforms all other methods w.r.t. $7$ different error measures and $20$ datasets.
 }
\label{fig:rank_changes_moe}
\end{figure}
\input{ekrem/algorithm-ranks}
\begin{figure}
  \centering
  \includegraphics[width=0.38\textwidth]{plots/mdmreValuesLOO.pdf} 
  \caption{Spectrum of MdMRE values for 2 regions of methods: Solo-methods and multi-methods. Here we keep the order of methods same as those given in \fig{algorithm-ranks} and divide those methods into 2 regions. Notice how multi-methods attain the lowest MdMRE scores.}
  \label{fig:mdmre-spectrum}
\end{figure}

\section{Results}
\label{sect:results}

\fig{rank_changes_moe} shows the rank of our 102 methods. As before, the x-axis ranks the methods according to number of losses
and the y-axis shows the ${\delta}r$ of each method. 
\fig{algorithm-ranks} shows the 102 methods, sorted in the same way as the x-axis of \fig{rank_changes_moe}.

Two aspects of these result are worth commenting:
\bi
\item The top $X=9$ methods (marked by a dashed line) are 
all multi-methods. 
The remaining multiple methods appear at ranks 14,15,18.
That is, in the majority case $(\frac{9}{12}=75\%)$, 
combinations of methods perform better that any solo method. Further, in all cases $(\frac{12}{12}=100\%)$, they are ranked higher than the majority
of other methods.
\item
The multi-method at $X=1$ also has the lowest ${\delta}r$ of any method in this study.  This method generated estimates using
the mean value of 13 top-ranked methods.
\ei
Note that the second result is exactly the ``ensembles are better'' result
as might be predicted by the researchers
mentioned in the 
introduction~\cite{Jorgensen2004,Seni2010,Hastie08,Kohavi95,breiman92}.
We speculate 
that prior SE researchers who failed to find that ``ensembles are better''
did not prune away inferior solo-methods before building an ensemble.

Not only do our results support the claim that ``ensembles are better'';
but they also support a related claim:  when building ensembles, then
``the more, the merrier''. \fig{algorithm-ranks}
shows that our best multi-method (at $X=1$)
was constructed from the most number of solo-methods (the top 13 solo-methods). 

Better yet, as shown in \fig{rank_changes_moe},  
this largest ensemble at rank $X=1$ had the lowest ${\delta}r$ seen in any of
our 102 methods (${\delta}r=1$).
This result underscores the main message of this paper: the method that scored the best (and had the greatest stability across different
experimental conditions) was the one that used the most number of \textit{superior} solo-methods.

%The two stand-out results from that are stand-out result from that figure is the method that appears at $X=1$. This method has the lowest 
%%In this section we firstly present the results of our research and then we map those results to particular research questions that were presented in \tion{research-questions}.
%We first evaluate our methods according to their total $loss$ values over all error measures and datasets.
%For that purpose, the procedure of  Figure \fig{WinTieLossPseudocode} was repeated for every error measures and dataset.
%Then $102$ methods were sorted by their total number of losses over all datasets, i.e. we expect the best method to lose the least when compared to the others.
%The resulting order is provided in \fig{algorithm-ranks}.
%The method, with fewest losses ({\bf Top13/Mean}) is ranked \#1 and the
%one with the most losses ({\bf PCA/LReg}) is ranked 
%\#102 .
%According to this order, 9 out of 12 multi-methods appear as the best 9 methods.
%
%When providing our results and answering the research questions, we would like to keep the ordering of methods as well as datasets fixed.
%This helps us better track the changes in different experimental settings.
%We will use the order provided in \fig{algorithm-ranks} for that.
%%The rank of all methods is given in \fig{algorithm-ranks}.
%In a similar fashion we also sorted the datasets according to total number losses.
%\fig{dataset-ranks} presents the sorted datasets from the lowest to the highest number of losses over all error measures.
%
%%\input{ekrem/dataset-ranks}
%

%After fixing the order of algorithms and datasets, we wanted to see how performance of individual methods would behave subject to different settings.
%%For example we are interested in seeing the performance of every individual method on every individual dataset.
%Now that we know the order of methods according to sum of $loss$ values over all datasets and error measures, we are interested in a more detailed analysis.
%In \fig{win-minus-loss-percentage}, each {\em x,y}  intercept corresponds to a method applied on a dataset.
%An intercept shows the result of $win-loss$ values of a method as a percentage of $707$ comparisons. Each method compared to 101 other methods using seven performance measures: \mbox{$101\times7=707$ comparisons}. 
%So the highest $win-loss$ value is $707$, where a method wins all comparisons.
%The {\em y}-axis shows 102 methods (sorted in the rank order of \fig{algorithm-ranks})  and the x-axis shows 20 datasets (sorted in the rank order of \fig{dataset-ranks}). 
%%For example, the top-ranked algorithm {\bf Top13/Mean} appears at {\em y}=1;
%%the {\bf log/ABE0-1NN} appears at {\em y}=22s; the {\bf log/LReg} appears at {\em y}=92;
%and the worst-ranked algorithm {\bf PCA/LReg} appears at {\em y}=102.



%Also note that \fig{win-minus-loss-percentage} shows two figures for two testing strategies: one for LOOCV (\fig{win-loss-loo}) and one for 3-way (\fig{win-loss-3way}).
%In both figures the order of methods and datasets are fixed as discussed before.
%Furthermore, in each figure the lines from left to right end correspond to multi-methods.
%
%When we look at \fig{win-minus-loss-percentage} we can see that there is a considerable uniformity on top ($y=80$ and upper) and bottom ($y=24$ and lower) parts.
%This tells us that good (bottom part) and bad (upper part) performing methods consistently appear within certain percentages of $win-loss$ values.
%%This uniformity means that high and low performing algorithms have uniformity across the datasets.
%Also note that including line $y=9$ and below (i.e. top performing methods) are the multi-methods.
%We can observe that multi-methods are among the successful methods, however it is difficult to distinguish them from other successful methods (in region $11 \leq y \leq 24$).
%Therefore, we will resort to the method of \fig{rank_changes}.
%
%
%\fig{win-loss-3way} is same as \fig{win-loss-loo} except the fact that it employs a different testing strategy.
%When we try to double-check the observations of \fig{win-loss-loo} on \fig{win-loss-3way} we see that the picture becomes more blurry, i.e. some of the observations are still valid they become less rigid and one observation becomes partially invalid.
%For example, the observation that \textit{``some of the datasets do not favor any particular methods''} now becomes partially invalid, in the sense that except telecom dataset, now all of these datasets have methods above $50\%$ $win-loss$ value.
%Also, when we look above method $60$ on y-axis, we observe that the methods still show low performance.
%However, this time this observation is less rigid, i.e. there are a bunch of methods above method $60$ that have more than 50\% $win-loss$ value.
%The congestion of high performance methods ($win-loss$ percentage above $50\%$) below method $20$ in \fig{win-loss-loo} is also observable in \fig{win-loss-3way}.
%But this time there are also a bunch of high performance methods scattered above method $20$.

%\begin{figure}[th!]
%  \centering
%  \includegraphics[width=0.45\textwidth]{plots/allLossValuesLOO.pdf}          
%  \caption{Loss Percentages. Number of losses seen in 102 methods and 20 datasets, expressed as a percentage of the maximum losses possible for one method in one dataset. The maximum number of losses for one dataset is: \textit{(102 methods - itself)*7 performance measures = 101*7 = 707} and minimum number of losses possible is $0$, so \textit{12.5\%} is \textit{707*0.125}, \textit{25\%} is \textit{707*0.25} and so on. The x-axis is the 20 datasets and the y-axis is all of $102$ the methods. The order of algorithms is the same as those given in \fig{algorithm-ranks}.}
%  \label{fig:win-minus-loss-percentage}
%\end{figure}

Other results offer yet more evidence for the superiority of multi-methods over
solo-methods.
\fig{mdmre-spectrum} sorts the MdMRE values of all the solo-methods
and all the multi-methods:
\bi
\item Multi-methods generated lower MdMRE values than the solo-methods;
\item
While some of the solo-methods have alarmingly large errors (as observed from the steep right-hand-side of the dashed line in 
that figure),
note that {\em none} of the $12$ best multi-methods have large  MdMRE values.
\ei
That is, multi-methods are far less prone to incorrect
effort estimates than solo-methods.

%Another alternative perspective of evaluation to MdMRE is Pred(25) values.
%In a similar manner to \fig{mdmre-spectrum}, we generate the 3-bands plot of methods for Pred(25) values in \fig{pred25_spectrum}.
%
%\begin{figure}[th!]
%  \centering
%  \includegraphics[width=0.48\textwidth]{plots/pred25ValuesLOO.pdf} 
%  \caption{Spectrum of MdMRE values for 3 bands of methods. Here we keep the order of methods same as those given in \fig{algorithm-ranks} and divide those methods into 3 bands. Notice how band1 (all multi-methods) attain the lowest MdMRE scores.}
%  \label{fig:pred25_spectrum}
%\end{figure}

%As a final sanity check, we report the PRED(25) values seen in the three regions of \fig{rank_changes_moe}.
%We perform this check since
%evaluation based on statistical hypothesis testing can misleading when the populations are significantly different, but have a very similar median performance.
%
%\fig{rank_changes_moe} shows the mean of maximum rank changes for all the methods in accordance with the procedure of \fig{rank_changes}.
%In \fig{rank_changes_moe} we observe that band1 (region of multi-methods) as well as band3 (region of low-performing methods) have relatively lower mean-change values.
%The mean-change amount for band2 on the other hand can go as high as 30.
%This scenario is similar to that of \fig{rank_changes}: High performers (in that case multi-methods) and low-performers are more stable than average-performing methods.
%
%One of the most important reading of \fig{rank_changes_moe} is that the most successful method {\bf Top13/Mean}, has a mean-change of 1, which means that among all datasets, and across all error measures, {\bf Top13/Mean} consistently performed as the best method.
%When \fig{rank_changes_moe} is interpreted together with the previous performance plots, we can say that multi-methods behave stably and perform consistently better than solo-methods acc-Ross multiple datasets and multiple performance criteria.
%

%Add in the discussion of showing different perspectives etc. Also refer to previous paper of method combination and tell that single perspective+multi-learner does not work, hence multi perspective with different pre-processors is very important!!!



