\newpage
\appendix
\section{Replies to Reviewer Comments}
\subsection{Replies to the Editor Comments}

Comments from Coordinating Editor

Many thanks for your submission. Attached are the detailed views of the reviewers;  based upon their comments, we would ask you to REVISE and RESUBMIT the paper at this stage. Please note, this does not guarantee that your paper will be accepted. While there is some variation in the rankings of the reviewers, they consistently agree on the issues with this paper:

1.	Presentation: in its current format, the paper is extremely difficult to understand. The paper needs considerable reorganization and the large volume of missing details must be supplied.

2.	Novelty: While the paper seems to be technically sound, it is far from clear that the paper has any real merit. What are the principle contributions of the paper? How does it advance SE knowledge? These must be clearly stated; if not, you should expect the paper to be rejected after the next round.

3.	In a similar direction: it is far from clear what impact or significance exists in the work? Why does your work have any relevance to the outside world? Or other organizations?

Good luck with these issues; we look forward to reading the next version of this paper,
\\

{\em We thank all reviewers and the coordinating editor for their time and effort on providing such detailed comments and guidelines for producing a better version of the manuscript. Regrading the concerns from all reviewers, we have re-organized the manuscript in the recommended way. Now the manuscript has a classical outline, starting with the conceptual motivation of the work, description of research questions, methodology (data, algorithms, performance evaluation), experiments, results, discussion, related work,practical impacts, threats to validity and conclusion. The results (and any related discussion) have been removed from the sections before the experiments are actually performed. We have added more details or removed some parts where the reviewers found necessary. For example, we have edited the abstract accordingly. we have written the introduction from scratch and provided details on the NN methodology. We clearly stated our contributions and reserved a separate section for tha implications of the study for software practitioners.\\}

-----------------------------------------------------------------------------------------

\subsection{Replies to the Reviewer \#1 Comments}

Reviewer \#1: Overall comment:

While I do have some concerns with both the form and content of this paper, my overall assessment of it is positive.  It reports work that appears to have been conducted rigorously, and it fits the interests of the intended audience of EMSE well.  My reservations relate to the conceptual motivation for the work, the true applicability of the prior 'related' work on effort estimation, and the impact of the lack of balance in the data sets.  I am also not comfortable with the way the work is presented, with results effectively presented before the experiments have even been described.  These are not minor issues, and some will require non-trivial effort to address - however, I think the core of the paper is sound and with changes it should be acceptable.  I elaborate below on the major points of concern.
\\

{\em We thank the reviewer for these comments. As we have explained in the Replies to the Editor Comments section, we have addressed all your concerns in this version of the manuscript. Below you may find our replies for the specific comments.\\}

Specific comments:

- Performance of WC and NN data is alluded to in the Abstract but it would be useful to actually have indicative values reported here - if they were actually very low then the fact that they were close would be of little consequence.
\\

{\em We have removed all results from the abstract and the body of the paper as all reviewers recommend. Please see new abstract.\\}

- Page 1 line 40: The evidence regarding generlity is not just theoretical - empirical evidence to date suggests that, at the project level, generality is low.
\\

{\em We have written a new introduction to address the concerns. In the new version the conceptual basis of the work is stated more clearly, also stated in the first sentence of the new abstract.\\}

- Page 1 line 56: Of interest to me here is the conceptual basis for the work.  What is the theory in SE that suggests that this might be (i) possible (ii) worthwhile?
\\

{\em Please see our new \S2.1.\\}

- Page 2 line 5: The question and response overstate the applicability of the work undertaken, in my view.  At most, two organizations are considered here, but the text gives the impression of a more generally held outcome.
\\

{\em We thank the reviewer for pointing out a discussion about the number of organizations considered. 
In the previous version we have discussed that NASA is an umbrella organization and the 7 projects used in this work came from different organizations. Therefore, we treat 7 NASA projects as if they were from 7 different companies. Considering the SOFTLAB projects, we analyze 8 different companies and a total of 10 projects. We have now included this justification in the text.
\\}

- Page 2ff: The ordering of material makes reading of the paper quite challenging.  We are here being informed of the outcomes of the experimental work, yet this work is reported much later in the paper.  I would prefer a more sequential style of presentation.
\\

{\em We agree. The manuscript has gone under rigorous reorganization.\\}


- Page 2 line 36: Use of the word "mere" reveals an important difference between this work and that previously conducted in relation to effort estimation, one that is likely related to both the conceptual justification for the work and the outcomes achieved; that is, the use of module-level data, of which there is far more when compared to the project-level data used in almost all cost estimation research. To me this issue needs more discussion as it impacts on likely homogeneity in data sets and also the feasibility of using WC models. Furthemore, defect prediction is based on the characteristics of a tangible artifact (the code) whereas project effort estimation is often design- or even specification-based.  As a result I'm not sure that we could even expect the effort-oriented work reported previously to bear any relevance to that undertaken here.
\\

{\em The reviewer is 100\% correct- our early mention of the effort estimation CC vs WC work was confusion and not required to motivate this paper or to understand its results. We have moved that entire dialogue into the related work.

}
- Page 3 line 9: Minor - incorrect spelling of "Intuitively".  Also note incorrect spelling of "Abrahamsson" and "MacDonell" to follow.
\\

{\em Fixed.\\}

- Page 3 line 43: While this is one possible explanation I suspect that the reasons are numerous.  Furthermore, some of these relate to experimental design as much as they do to issues of measurement.  Still, ambiguity is one possible reason that could contribute.
\\

{\em We agree with the reviewer. We do not explore other possible reasons, since they would not contribute to our justification. The explanation about ambiguity in features, we believe, justifies our use of static code features \\}


- Page 3 line 54: The comments re the lesser ambiguity of static code features need support.  Some of these measures can still be interpreted and/or collected differently - although I agree that the degree of ambiguity is likely to be less than for some other features.
\\

{\em That whole dialogue on CC for effort estimation was, as the reviewer quite
rightly says, a waste of time. We have moved it to the back and removed
all mention to the issue of feature ambuiguity .\\}


- Page 4 line 11: "For the two experiments..." Two does not seem to be the right number - is this 'left over' from the prior literature review?
\\

{\em Fixed.\\}


- Page 4: The discussion of resource management in relation to defect prediction is based on a fairly coarse model (defective/defect-free) - no sense of value of each module, which might also be considered in determining which should be inspected and when.
\\

{\em 
We agree that the priority/ value of each module has different impacts on resource management. We do not perform a value based analysis and our data sources do not consistently include this information. Rather we assume there is no distinction between modules. Thanks to your comment, we now discuss this issue in the threats to validity section. We also include the estimated testing efforts in practice, following the guidelines provided by Arisholm and Briand.

Also, please see \S2.2
\\}


- Page 4 line 34: Again we encounter discussion of outcomes of the current work presented here in the Related Work section, creating (for me) some ordering issues.  (See also page 6.)
\\

{\em This issue is fixed with reorganization of the manuscript.\\}


- Page 4 line 39ff: These are interesting reflections but how robust are they?  What is the empirical evidence base?  Furthermore, as this paper itself addresses, there is more to being 'Useful' than just achieving high detection probability.  Also, my interpretation of the description for 'Easy to use' suggests that this is actually indicating features that are 'Easy to collect'.
\\

{\em We agree about the issues. However, these reflections are based on a panel discussion in IEEE Metrics 2002 and a personal communication as clearly stated. There would certainly be variations in different applications. We found it feasible to report the experiences and educated guesses of noted researchers, who are involved in these communications, which are also published as part of a e-Workshop proceedings. We also include more work from literature for the empirical evidence base.  \\}


- Page 5 line 28: The one-at-a-time listing of papers that have used static code measures feels a little pedestrian to me, and serves to add many papers to the reference list that otherwise contribute little to the work reported here.  I'd prefer to see this list pruned.  \\

{\em We agree. the list is pruned by including only the most recent publication of each researcher in the list.\\}


- Page 5 line 32ff: While interesting I'm not convinced that such personal anecdote is appropriate here.  \\

{\em We mention it now at the start of the paper to motivate the work. We take care
there not to use it as evidence for some proposition- only a gentle introduction
to encourage the reader to ease into the paper. We can remove it,
if the reviewer recommends it. \\}


- Page 5 line 51ff: It is important to ensure that the context for the criticisms of static code features is made clear.  As predictors of development effort they may well be inadequate - however, for the purpose of defect prediction their effectiveness could be expected to be better. (See also the next two points.)
\\

{\em The criticisms for static code features are intended to be for defect prediction. We have reworded related parts to make that distinction clear. Our purpose in this part is the justification of our use of only static code features and not the other ones (i.e. process, team, change history features) \\}

- Page 6 line 7ff: We are here informed that the prediction is in fact a binary one - up until now my expectation had been that - as for most cost estimation work - the prediction would be of defect count or something similar.  This is another important aspect that distinguishes it from prior work on effort estimation, and again would bring into question the relevance of expectations derived from that prior work.  This would be like predicting effort into two very unbalanced bins e.g. 'typical effort' and 'atypical effort'.  Whether prediction of such a classification would be better than that reported for actual effort values is unknown.  (Note that a comment is also made in the text about the predictors out-performing simple thresholds on lines of code, but this is not referred to subsequently in the paper even though it is said to be untrue.)
\\

{\em We take care to discuss the binary nature of our predictions very early
in the paper (see \S3). Also, we have removed the effort estimation dialogue
from the front of the paper where it conflated too many issues.
} 

- Page 6 line 7ff: Also stated here is an expectation that a predictor would perform badly if based only on static code features.  However, given the unbalanced nature of the data sets, most data miners will get good pd values - by predicting that many are defective this will catch most of the actually defective modules (as occurred here) but will create many flase alarms (as also occurred here).  Similarly, predicting many as non-defective will produce a relatively low number of false negatives, as most are indeed non-defective.  The results obtained are therefore not hugely surprising to me - and some reference to prior work on unbalanced data sets, the use of over- and/or under-sampling and so on, should be included here. (In fact the latter may make a useful follow-up study by the authors.)
\\

{\em 
That is why we do not use performance measures like accuracy and precision
which, previously, we have shown are most uninformative for these kinds of database.

~\\
\begin{tabular}{c|cc}
what&non-defective&defective\\\hline
detector silent&A&B\\
detector triggered&C&D
\end{tabular} 
~\\

The reviewer is pointing out the issue that when $(A+C) \\gg (B+D)$, then
some performance measures are confused. eg. accuracy=$\frac{A+D}{A+B+C+D}$ and
you can get high accuracyies with large $A$ values. But probability of detection
(a.k.a. recall) is $\frac{D}{B+D}$ so it is possible to have high PDs in
unbalanced data sets (indeed, we achieve PDs over 70\% in most cases, despite
working with unbalanced data sets).
\\}

- Page 6 line 22: Minor - confusion between Figure and Table.
\\

{\em Fixed.\\}


- Page 7 line 58: Even in terms of defect detection the binary classification is very coarse - I do not agree that the work addresses methods for detecting errors, rather it indicates only whether or not a module is defect-free.
\\

{\em We re-worded that sentence as your comment suggests .\\}

- Page 8 line 21: The value of the remarks regarding the Mozilla work is limited here.  I recommend that it be dropped.
\\

{\em Dropped.\\}


- Page 8 line 57ff: With respect, I think to extrapolate from the NASA data repository to "the general software engineering industry" is rather fraught.  For one thing, what is "the general software engineering industry"?  Software is so pervasive now that such generalisations must be questioned - for instance, do NASA-related outcomes apply to wearable computing applications? to location-aware services? to software gameplay? to ERP-like developments?
\\

{\em We agree that it should be avoided to interpret this as a fact. However, we would like to keep this part, which is based on noted researchers recommendations and as our experiences (some reported in the paper) are consistent with this issue. For instance, in this paper we observe the same trends in washing machine software as in mission-critical flight software. Further, we observe the same patterns (not reported in the paper due to confidentiality) in a large telecommunication system.\\}


- Page 9 line 56: Should Treatment 1 not refer to 6 tables, rather than 11?
\\

{\em Thank you for pointing out, this has been fixed.\\}


- Page 10 line 44: The decision not to consider the languages used surprised me - along with other information regarding, for example, degree of reuse.  I would have thought that such data might have added to the robustness of the analysis, as well as the effectiveness of the static code features some of which will be affected by the language used.  (I would be surprised if a practitioner would be prepared to use data from a source that they knew virtually nothing about, or would assume that results obtained from C programs would apply without amendment to JAVA programs.)
\\

{\em We are surprised in this result. but it does indeed hold. in a  forthcoming report, we discuss an experiment which explored the effect of language on defects. Interestingly enough, other factors far out-weighted language choice
w.r.t. introducing errors.
\\}

- Page 11 line 29: "All the numeric distributions in the Figure 2 data" is a rather vague descriptor - needs clarification.
\\

{\em This part has been edited accordingly during reorganization of the manuscript. While describing the Naive Bayes model, after we introduce the Gaussian distribution, we discuss that all project data are numeric that have skewed distributions and hence we use log-filtering.\\}


- Page 11 line 37: Did the work reported in [8] also consider the SoftLab data?
\\

{\em No, because that data was not publicly available at that time. This is now mentioned in the text. As a matter of fact Softlab data are collected during the efforts to produce this manuscript.\\}


- Page 12 line 53: It is indicated here that the median value for the percentage of defective examples is around 8\%, but I cannot see how this is possible given the values reported in Figure 2 (with only two values less than 8).
\\

{\em This was an error in our part. We now fixed it (median value 13)\\}


- Page 13ff: Given the unbalanced nature of the data sets a summary of the actual values for the elements of the pd and pf measures would make for interesting reading here.
\\

{\em Considering the number of our experiments we do not find it practical or feasible for reporting individual numbers for each experiment. However, we are ready to include these numbers as an appendix if you consider that it is really necessary.\\}

- Page 13 line 43: What is/was the conceptual basis for the pruning of the tables to a common set of predictors?
\\

{\em Not conceptual: pragmatic. If we'd reported these results using different features, we suspect that 
right now we'd be writing a sentence to a reviewer justifying why we didn't prune to a common set.

We now included a new table showing the features-attributes included in each project. Since we build a cross company model in this experiment, the input features of the model should be available in all projects used for training. All 40 features listed in that table can not be used for building a model, since some features are not available in certain project's data. Therefore, we have used a common set of features that are available in all projects. 

Also we have changed the notation of terms throughout the manuscript for consistency. We were referring to projects as tables in some parts of  the previous version, which we think introduced some ambiguity. We fixed the notation in this version.   \\}


- Page 14 line 22: Offering readers the chance to skip a section addressing validity does not sit comfortably with me! (See also page 20.)
\\

{\em Fixed.\\}


- Page 14 line 34: I'd have thought that the fact that the only item in group b was the JAVA system warranted some comment.
\\

{\em That group never stood out in our results. We suspect we know but that is another paper in preperation
(hint: programming languages change but what is common in all these systems is that a {\em human}
wrote it.   \\}

- Page 14 line 58: Why was this detector not used here?
\\

{\em This part has been removed, since it was related to the personal experiences that are not found appropriate for mentioning in the text.\\}

- Page 15 line 47: Evidence for the assertion that most sites do not accept high false alarm rates is needed.
\\

{\em We now talk about experience with our user groups.
We now discuss this issue considering the study by Arisholm and Briand. We describe the circumstances why such high false alarms are not accepted. Please see the new practical implications section\\}

- Page 16 line 59ff: The NN-informed results have very limited ranges.
\\

{\em Thanks for pointing out. This shows that NN results are very stable in deed and words on this are now included in the text. 
\\}

- Page 18 line 39ff: Just because modules in a CC data set are similarly characterised is no guarantee that they are taken from a single company.  Furthermore, a single 'company' may have many different Divisions, may be multi-national, and may have practices that are more diverse than those seen across several companies in a single domain.
\\

{\em While we agree with you, this part is removed due to another reviewer's comments.\\}

- Page 19ff: A reference to prior work on transductive learning would seem appropriate.
\\

{\em We looked, but could not find anything immediately relevant. Do you have a reference? \\}

- Page 20 line 47ff: Or there are other forces at work that constrain further learning e.g. thresholds are enforced so that modules are not longer than a given length.
\\

{\em The limited information content refers to another published publication and is a possible explanation for the observation in this part. While we agree that there may be other factors, we did not want to speculate and did not find relevant to include in the text .\\}

- Page 21 line 4: Two to four months is the duration for one person working on the code - this may be reduced further if more than one person takes on the task.  So the acquisition of WC data may occur even more quickly.
\\

{\em Thank you for pointing out. Later in that part, we mention that these numbers are person-month. We now fixed it in its first appearance.\\}

- Page 22ff: A general comment relating to all of the experiments - this seems fairly typical of a data mining paper wherein the actual predictors, their influence and so on, receive absolutely no discussion at all - the focus is entirely on classification accuracy.  While accuracy in undoubtedly important it is not the only desirable or interesting characteristic of a useful prediction system - and this should be acknowledged in the paper.
\\

{\em We discuss about the method of choice (Naive Bayes) by referring to other studies that found it useful briefly in the method description and in detail in the threats to validity section.\\}

- Page 23:  Are the SoftLab data sets (being smaller) simply more homogeneous?
\\

{\em No- the ranges seen in them have broader references.\\}

- Page 25ff: The form of the following references is incorrect - [3], [11], [19], [40].  Also references [37] and [59] are effectively duplicate.
\\

{\em That style was given to us by IEEE. Perhaps we could fix it with help from the production staff? 
As to the repeated entries, could you check those numbers please? We cannot see that on our current draft.



~\\}



\subsection{Replies to the Reviewer \#2 Comments}
Reviewer \#2: 
General comments

If my understanding is correct the nub of one of the propositions in this paper is that by selecting similar projects from a CC data set to those that are present in the WC set it is possible to predict modules that contain errors. This does not seem to be a surprising or novel result if we select the projects carefully. The real problem would appear to be how do we decide what is similar data when we have very few if any projects in the company's set of past projects: what should we consider as a similar set of data, whether we use a nearest neighbour (NN) algorithm or something else, and, therefore, in general, for the case in which we have CC data how should we define a nearest neighbour for any given company's data set. That is, how do we define an appropriate subset of the CC data? This is an important point that the authors should address clearly. 
\\

{\em We thank the reviewer for these comments. Although the reviewer suggests that the results are not surprising or novel, there are no prior publications in defect prediction literature that investigates this issue. However, we agree that we should have more clearly stated our NN selection criteria. This is now included in the text and also in the new experiment design pseudo-codes. Our similarity measure is simply the Euclidean  distance between the data points in the feature space, which is the space of static code features. Also, we have some evidence that our selection criteria was effective (recall that it reduced false alarms
and increased PD.  \\}

It seems clear that different developers will have developed the software in WC and CC data and perhaps even different development strategies may have been used resulting in different module design within software and a variety of erroneous modules. Some developers may have used CASE tools and others not etc.. So these attributes are unlikely to be common between WC and CC data. Hence, the authors must state in their paper (preferably in a Table or the Appendices) what attributes they have used in their experiments in order to derive their NN algorithm, at present they don't do this.
\\

{\em Thanks for your comment. One of the points of this approach is that it {\em is} possible to automatically collect a uniform set
of measurements on source code using feature extractors over static code attributes. It is for this reason
that our colleagues at NASA IV\&V run these feature extractors over all examples from their domain.

But we agree and we did not make that case adequately.
We have now included a table of features available in all projects we have used. 

We have also marked the features used in different experiments, including NN, and described all experiments in pseudo-code form. As we have mentioned above, the similarity measure is the Euclidean distance in the feature space of static code features given in the new table, and this is clearly stated in the related experiment design part.  \\}

Further, it seems obvious that if we have enough WC data then it is much easier to select similar data from a CC data set and hence get some good 'predictions' of what has happened in the WC data. So it is difficult to see what the novelty of this work is. \\

{\em Thanks for your comment. To the best of our knowledge, no one has 
rigorously explored the value of using alien data on the local projects. 
Yes, certainly k-NN methods have been used before but not, as far as we know, as a fix to the problem of
importing alien data. We now discuss this in the related work section.\\}

Further, another proposition is that using Naive Bayes and some well known metrics that we can predict modules with errors this doesn't seem novel either, partly because the metrics were intended to be used for such purposes (or were expected to be related to numbers of errors in a module). It is also difficulty to see the novelty in terms of the metrics that appear to give success. Further naive Bayes is a method that has now been used quite often in SE work.
\\

{\em Thanks for this comment, since it provides us the opportunity to better explain the novelty of our work. Since our 2007 TSE paper is the current highwater mark
in documenting the  benefits of Naive Bayes for defect prediction,
we must agree. But the innovation here is not ``use of NB''. Rather, it is ``use of alien data''.

We also recommend that if a company has enough WC defect data, it should use it without considering CC data. However, the core idea of this paper is intended for companies that have no WC defect data. While it seems obvious, we should again note that there is no prior work on this topic. This is one of the points for the novelty of our work. For your comments on the metrics and Naive Bayes method used: we should clearly state that our work is not a 'new metric suite' or a 'better predictor' study. Rather we want to provide an analysis for companies (with no local defect data), so that they can easily start employing a defect prediction model. Therefore, we have selected these commonly used methods and tools in our WC-CC analysis, so that it can be easily employed. Additionally, our results suggest that for mission critical projects, CC models can predict more defects than WC models (with some cost of false alarms), yet still avoiding exhaustive testing and detecting more defects. \\}

Clearly in order to produce a novel paper that will be useful to others the authors need to discuss the originality of the method they propose. However, since the probability of false positives is quite high, for the method to be of value, they also need to illustrate how the method they have developed could be improved so that the false positive rates can be lowered considerably. Otherwise there seems no point in using the approach: a software engineer may as well examine every module for errors. For example, for CC data (?) and if I have understood your description of pd and pf, if pd is 97\% and pf is about 64\% and if you are looking at 200 modules you need to examine 161 of them to find 97 erroneous ones (assuming that there are lot of modules with errors, i.e. 100 out of 200). (A potential scenario could then be that if the engineers find 90 plus modules with errors they stop looking at the remainder of the 161 modules because they probably won't have errors.) Even if
there are only 20 modules and 11 with errors then this implies that you would need to look at about 15 out of 20 modules and you are still likely to miss one module with errors. So why not just look at all of them anyway? The circumstances under which the authors state the prediction system to be useful are not convincing and will not occur often (as they do indicate). I believe that the importance of what a nearest neighbour should be is fundamental to the acceptance of this paper, particularly when there are very few or no data in the company's project data base. This needs to be discussed and evaluated at length and should include sensitivity analysis. It isn't and doesn't, at present. In addition, all the classifiers need to be identified for the data used. Some form of generalisation needs to be given which indicates what classifiers might be useful for other companies. (Or are you saying that the classifiers used are always only LOC, McCabe and Halstead exclusively and
each time all 21 values in your Fig 3 are used? You need to clarify this, because this could restrict application of the method.)
\\

{\em Let us start from the last issue. Yes, we use only the features given in (previously) Fig3. We already discuss in the paper that other features (process, team, change history etc.) can also be used (and should be used in further studies). The reason for using these, as explained in the paper, is that static code features are the most basic and easy to achieve data from different companies. And our data sources, e.g. NASA projects, contain only these features and have no further information about the process or team related features. In order to use a CC model, one should collect data from different companies in the same format, which is difficult due to confidentiality. However, once again, static code features are easier to collect from different companies or available public repositories. 

About your concerns on the quality of predictions: we agree that the false positive rates in CC models are quite high, and we clearly state that this prevents the use of CC models in general. This is why, we propose the NN-CC model for lowering the false positive rates, in which we have success. Our analysis shows that NN-CC models can be used as a viable alternative in the absence of local (WC) data. Once again, WC models are better (as expected), however our analysis proposes a guideline for practitioners that allows them to perform defect prediction instantaneously  without collecting WC data. We think that this is another point for novelty and impact of our analysis. As, we mentioned above, we agree on your comments on the selection of neighbors and we have detailed it in this new version.

The defect rates, as seen in the projects in NASA and SOFTLAB projects and as our personal experience suggests, are not around 50\% and much lower in deed. This issue is mentioned in the paper. Let us consider MW1 project in the paper, since it has similar performance rates with your example. For MW1, there are a total of 403 modules with 31 defective and 372 defect-free ones. CC model yields 90\% pd and 68\% pf, and one should examine 280 modules, which is around a 31\% reduction in inspection efforts compared to examining all modules. Still, we argue in the paper that 68\% pf rate is quite high and using NN we are able to reduce it to 33\% pf and 68\%pd, which corresponds to examining 144 modules, a reduction of 47\% compared to exhaustive testing (and we assume an exhaustive test should examine 274 modules for detecting 68\% defects, as Arisholm and Briand suggests). Thanks to your comment, we now include this analysis in the manuscript.  \\}

The paper does not use an appropriate format for ESE - it reads like chapters that have been cut from a draft PhD thesis. The overall structure needs to be reconsidered to give a paper that flows more readily. Much of the introduction needs to be removed and some needs to be reconsidered., e.g. discussing the results before describing the data the models and experiments is not helpful, P2 onwards and a discussion of effort estimation data is not relevant. Also presentation of the data using quartile ranges is not informative enough. The actual pd and pf values should be given, in a table or an appendix.
\\

{\em All this issues have been resolved in this revised version. However, we think that quartile charts (similar and more informative than boxplots) are useful for reporting results. They have been used in prior publications and provide more than reporting only means and standard deviations. Boxplots clearly show the distribution of individual numbers. Considering the number of our experiments we do not find it practical or feasible for reporting individual numbers for each experiment. However, we are ready to include these numbers as an appendix if you consider that it is really necessary.

We agree with the reviewer that our early mention of the effort estimation CC vs WC work was confusion 
and not required to motivate this paper or to understand its results. We have moved that entire dialogue into 
the related work. 
   \\}

Specific comments

Page 1 line 16 There are 3 experiments I think, state the number.
\\

{\em Abstract is rewritten without the results. Please see new abstract .\\}


P1 l26 not sure that stop-gap has a genuine meaning here a more appropriate term could be found.
\\

{\em During re-writing, we avoided the use of this term. \\}


The English usage is a bit awkward to read in places and should be improved throughout but L39 is an example.: asking 2 questions at the beginning of the introduction and giving only one answer "no" is not helpful. I am not sure that there is a theoretical answer to these questions anyway but if there is then a reference is needed.
\\

{\em Introduction section has been rewritten addressing the concerns of all reviewers.\\}


L 53 Again, how should this pruning be done? There is a problem here once you have selectively deleted data from a data set it is no longer a random sample from the original population (assuming that it once was). It is now biased, so any statistical theory you choose to use such as naive Bayes to produce your models becomes at best an ad hoc solution and may not be valid for other data sets (since you may not be able to identify the sub-population) and may not be valid for another random sample from the original population of data. You would need to quite clearly state what the sub-population was that you have selected. Hence, my comments about how NN was done.  Further, it would appear that the sub-populations could be different for different companies. 
\\

{\em We agree that sub populations are different for different companies. Since we now explain the method of selecting the sub populations from CC data in NN part (i.e. Euclidean distance in feature space), we think this argument is now clear. However, we have removed the reference to that work during reorganization \\}



L5 p 2 is an example of the incorrect format used for headings (in this journal) throughout this paper
\\

{\em All formatting now complies to Springer journal style.\\}


L7 - You are trying to justify the results before you have discussed the problem and its background.
\\

{\em Resolved this issue during re-organization.\\}


L12 P2 deprecates must be the wrong word here, is it?
\\

{\em removed from introduction, fixed.\\}


There are quite a few problems with incorrect English usage, idioms and colloquialisms that need to be corrected before publication. E.g. L20
\\

{\em We took great care in this version.\\}



Words like fluke need to be avoided and appropriate scientific terms used.
\\

{\em We agree, we did our best to avoid such terms.\\}


P2
I can see no reason why so much space is wasted discussing effort estimation data. It is superfluous. Unless the authors are indicating that effort estimation prediction and defective module prediction are in some way linked and is used in some part as a classifier? 
\\

{\em Prior to this manuscript there were no studies investigating the WC-CC issue for defect prediction. We included the discussion about effort estimation studies to show that this issue is not as simple as commonly perceived. For instance, there is a common belief  that local data is always better. What we intend to show is that at least for effort estimation this common belief is not correct. There are many diverse conclusions. Therefore, we try to justify our intention that such a WC-CC analysis is required for defect prediction. Although our results turn out to be expected, we believe that we fill a certain gap in the literature and we also propose practical ways of benefiting from CC data. We are only interested in the conclusions from using either WC or CC data (i.e. WC is better, CC is better, neither is better). These conclusions are not clear in effort estimation, however we find clear conclusions for defects prediction

Nevertheless, we agree with the reviewer that our early mention of the effort estimation CC vs WC work was confusion 
and not required to motivate this paper or to understand its results. We have moved that entire dialogue into 
the related work. 
\\}



L38 examples of what?
\\

{\em Examples of data samples (i.e. size of training set). Removed this discussion about results.\\}


Very weak English throughout this page (2)
\\

{\em Fixed.\\}


What is very little data? L40 P2
\\

{\em We meant small amounts of data. Removed this discussion about results.\\}


P2 is very poorly written. 
\\

{\em Fixed.\\}


P3 seems almost totally irrelevant and also does not support the argument about CC and WC data, i.e. there appears to be no contradictory results here. Where evidence exists,
 WC is best!
\\

{\em Please see our above reply for the inclusion of effort estimation studies .  \\}


P4 clearly state what the hypothesis is that you are testing
\\

{\em Hypothesis is misused in this part. Removed.\\}


P4 L23 No justification for using a prediction system is needed, provided it predicts a future event.
\\

{\em Removed.\\}


P4 seems superfluous
\\

{\em Most parts are removed with your suggestions (like your above comment). \\}


If you are going to discuss pd and pf here you need to first define them and give an example of how to calculate them or just leave their definition until it is needed during analysis and experimentation and then give an example.
\\

{\em Resolved  with re-organization.\\}



P5 also seems largely superfluous static analysers are used, we know. Your work seems to support their value: unless you are saying your features need to include other non-static metrics in order to work?
\\

{\em Our purpose in this part is the justification of our use of only static code features and not the other ones (i.e. process, team, change history features). However, we have removed the part about the static analysers as per your suggestions. \\}

P6 strictly most of your figures are not figures  - they are tables. But again you are discussing the results in the introduction.
However, you do need to tell us here that you are using NB and about how they learn about predictors (classifiers) and you don't as far as I can see.
\\

{\em We thank the reviewer for pointing out this issue.  Figures and Tables are labelled correctly in this version. Results are removed from the introduction and description of Bayes model is given in the Methodology part\\}

P7 much of this can be removed.
\\

{\em Removed all p7 content after (previous) Data section.\\}


P7 L7-21 You need to consider whether this is really relevant at this point.
\\

{\em We have moved this part to Practical Implications section.\\}


P7 L47-48 [42] you need to give more detail in this paper since it could directly affect how you define your NNs
\\

{\em We have removed this part as per your above comments and since feature selection is not considered in the current work. However, as mentioned earlier we have described the features and similarity measure in NN more clearly.\\}


P8 L1-10 This demonstrates a lack of feature not the presence of features! What were the projects' features ?
\\

{\em That part is removed and a new table showing the features available in each project is included  .\\}


Again P8 has many irrelevancies in it. E.g. Debates etc
\\

{\em Those parts are removed and other text are moved to Data and Threats to validity sections.\\}


P8 L21 Don't tell us what you started and didn't do -we don't need to know this.
\\

{\em Removed.\\}




P9 L41 why in reserve  - you use them don't you?
\\

{\em They are 'initially' in reserve, we use them later for external validity.\\}


P9 L55-57 You need to say what the treatments are. These are data sets or sub sets aren't they? In what way did you treat the entities before taking the measurements? (If you did.) How is treatment 1 different from treatment 2 or is there really no treatment involved? 
\\

{\em This is an error on our part. Treatment is not the correct word. We avoided using it and explained the experiment design in pseudo code for each experiment. Our intention was to refer to WC experiment and CC experiment. That is "Treatment 1" was CC experiment and "Treatment 2" was WC experiment.\\}


P9 L49 This experiment is badly described.
\\

{\em We have moved most of the content to related parts in the reorganization of the manuscript. This section now only focuses on the experiment design and supported with a table describing the pseudocode of the experiment. We believe it is more clear in its current form.\\}


Figure 4 \& 5 are not figures they might be Appendices or part of a Section if they need to be described.
\\

{\em Thanks for the comment, we have removed these considering that they are well-known by the intended audience of the journal.\\}


P10 L54 badly explained, re-write.
\\

{\em Fixed.\\}


P11 L 29 How do you know that these data are exponential?  What kind of exponential distbution do they belong to? Do you actually mean they are not symmetric and are skewed?
\\

{\em This refers to a prior IEEE TSE publication and we did not want to repeat content. This part has been edited accordingly during reorganization of the manuscript. While describing the Naive Bayes model, after we introduce the Gaussian distribution, we discuss that all project data are numeric that have skewed distributions and hence we use log-filtering.\\}



P11 L37 What do you mean by best - define it.
\\

{\em We mean in terms of performance measures we use. Re-worded.\\}


P11 40-41 Don't refer to unpublished and incomplete work it hasn't been peered reviewed and may be incorrect.
\\

{\em We agree, removed.\\}


Most of P11 is not relevant to the paper propositions.
\\

{\em Removed the parts related to personal experience and unpublished results. Other parts are moved to related sections (i.e. threats to validity)\\}


P11 Figure 6 doesn't seem relevant and the title is ridiculously long.
\\

{\em the figure is moved to the threats to validity section, it allows us to justify the use of Naive Bayes model providing empirical evidence. It is from a recent IEEE TSE publication by Lessmann et.al which makes extensive experimentation and we beleieve it shows that our Bayes model choice is a feasible selection. This figure also addresses other reviewers concern on mentioning the pros and cons of using this model. We agree that the title is too long and we have fixed that. \\}


L55 - 56 Give an example of how to calculate pd and pf so that the reader knows how you derived your numbers and how to do the same.
\\

{\em Thanks for the comment. As mentioned in our replies to your general comments, we now include an example, however in a later part\\}


P12 L45 - 46 explain what balance is supposed to show and give an example here
\\

{\em Added .\\}


P12 L57-58 You also need to show the actual values for pd and pf. I don't think that the quartiles are particularly helpful you are in fact summarising and loosing information.
\\

{\em Please see our reply for your general comments.

FY: we have generated those graphs and they look the same. Our view is that they add little to the discussion but if the editor permits a larger version of this paper we would be happy to add them
\\}


Fig 7 is not a figure. It needs to be a Section, and what is your kind of data ? It isn't clear why you need to give us a formula for the Gaussian distribution explain why? Are your feature data now normally distributed?
The paras at the end of the figure don't seem relevant?  Why are they? If the so called events are correlated as you suggest then how badly does the lack of modelling of this correlation effect the results? Sensitivity - could this explain your high pd and pf values or not?
\\

{\em This part is moved as a subsection in the methodology section and irrelevant content is removed. As mentioned previously, while describing the Naive Bayes model, after we introduce the Gaussian distribution, we discuss that all project data are numeric that have skewed distributions and hence we use log-filtering to transform the data for better goodness of fit. \\}


Figure 9 does not convey much meaning it needs to be explained and it is a Table!
\\

{\em We have moved this to the related part in the experiment discussion where it is explained and marked as a table.\\}


P13 Explain why you are using the Mann-Whitney Test you have not defined what your treatments are - explain how you used M-W T and why. Re-write this. What about the pf values do they differ? 
\\

{\em We directed the readers to the article by Demsar on this issue. However, we have extended the discussion on why we have used it. We have used this test for all performance measures and this is now clearly stated in this part. \\}


P13 L10-17 Clearly state your hypothesis it isn't clear what you intend to show - how do we know what is irrelevant and why? This needs to be clearly explained.  How do we know what is a nearest neighbour?
\\

{\em This discussion is edited accordingly and a formal hypothesis is described in the NN experiments in the new version of the manuscript.\\}


P13 L41 what do you mean by a prior procedure? Is this Bayesian? What do you mean?
\\

{\em We meant the experiment design. We have edited this part snd ee now explain our design more clearly using pseudo codes.\\}


P13 L60 Recall What? recall from what section and which previous result? It isn't clear at all.
\\

{\em We agree. Fixed.\\}


P14 L8 If pf = 100\% it would mean that your method failed to correctly classify any modules without errors, meaning that all modules were classified as having errors except those that were misclassified as having no errors (that in fact had errors). Is that what you mean? Clearly this is not good. Explain how this could happen and why you have achieved 100\% and what it means to a software engineer.
\\

{\em We certainly agree this is not good and In this part we wanted to elaborate this by reporting this maximum pf rate. However, this extreme case can not be interpreted as this approach is not useful, likewise we can not interpret the minimum pf rate (i.e. 14\%) for supporting the approach. With your permission, we have included similar text regarding your concerns. \\}


P14 L12 what is an irrelevancy and what do you mean by numerous, what do you mean by statistics:  averages, standard deviations or something else? Do you really mean attributes or features? State the hypothesis clearly it isn't obvious what you are saying.
\\

{\em This comment has been addressed earlier.\\}


P14 L21 use a less colloquial title and correct format
\\

{\em Done.\\}


P14 L23 no it can't be skipped (colloquial) it is highly relevant!
\\

{\em Thanks. we agree.\\}


P14 L32 what is the general pattern and of what? Be specific.
\\

{\em Thanks for pointing out. The pattern is : both $(pd,pf)$ increases in CC models compared to WC models. Fixed in text.\\}


L42-L57 all three points simply mean you could (and probably should) examine all modules. Why bother with your method? What advantages does your method have over examining all modules?
\\

{\em The advantages over examining all modules is discussed in the practical implications section. We agree that these three points suggest examining all modules but we also discuss that it is only valid for certain domains, such as mission critical applications for the first point . However, ability of detecting most of the defects without exhaustive testing would clearly be preferred and we demonstrate this in the new practical implications section. \\}


P15 L 38 This detector trigger is not relevant to your study - unless it is a feature of the NB model. Is it?
\\

{\em Removed.\\}


P15 L 61 Again why is effort estimation relevant to your study?  To P16 L21 This is not a sentence. This lacks relevance.
\\

{\em Please see our previous reply on the effort estimation. We have edited this part removing the irrelevant sentence\\}


P16 L32- You need to give much more detail about how NN was applied, what attributes or  features of the data were considered to represent the company data within the CC data. Particularly you must say how you derived theses features and what could be done when there is no company data in-house. This is fundamental to your thesis and without it no one can replicate you experiments. 
\\

{\em We have now clearly stated the similarity measure and the features used in the calculations as stated in our earlier response.We also mention in the text that these features are collected easily with automated tools and for our method no related defect data are necessary.\\}


P16 L41 -49 This needs to be re-written clearly it is ambiguous. What is a similar sample?
\\

{\em Same as above.\\}


P16 L54-57 I don't think you need to repeat a justification for using NN here it is obvious.
\\

{\em Removed.\\}


P18 L 8-9 This statement just implies that you have a poor method of discriminating between modules with error and those without. What evidence do you have that in general you can't have high predictive capability without high false alarms give the reference - or are you just talking about your own data?
\\

{\em Empirically, this result was seen in many of our 2003 papers discussing this data.
Mathematically, we showed that it was the general case in our recent TSE article:
	Problems with Precision,
	Tim Menzies and  Alex Dekhtyar and Justin Distefano and Jeremy Greenwald, IEEE TSE Sept 2007
	\url{http://menzies.us/pdf/07precision.pdf}. 

.\\}


P18 You must give more information in your BBN model and elsewhere of how the predictors or classifiers were chosen. (features?). I assume you didn't just always use all values for your three basic metrics?
\\

{\em We have clarified this issue. We use only a common subset of static code features (19 out of 40 features) in our experiments, which are mainly grouped into three: McCabe, Halstead, LOC  \\}


P18 30-31 Say what you mean by NN sampling (since you presumably must chose attributes it can't be random can it? Explain how you do it and what do you mean by random sampling in contrast.
\\

{\em We have used the term filtering in this version rather than sampling. Our attributes are fixed (i.e. 19 common attributes in all datasets), and by filtering(sampling) we mean the sub-populations used to train the Bayesian classifiers.  \\}


P18 L39 Just how much do we really know about future modules - you need to state what the features are that we know will occur in the yet undeveloped systems. Then the reader can decide whether your approach seems realistic and they can decide whether it could be improved upon and used.
\\

{\em As mentioned above, these features are static code features that can be immediately extracted after the source code is implemented and before it is  tested.\\}


P18 L 49 3rd word Which results?
\\

{\em Results from our experiments. Fixed.\\}


P18 L57 what do you mean by examples here? Clarify.
\\

{\em We mean data samples. Fixed..\\}


P19 L 36 What do you mean by adequate - define this.
\\

{\em We mean defect predictors that are helpful in practice. However, we now find this word pointless, since defect predictors are inherently intended to be helpful in practice. Therefore , we removed this .\\}

P19 L 43 Say why this is curious - I would say not because I think you are using metrics that were supposed to be designed for this purpose!
\\

{\em Fixed (important).\\}


P19 L48-52 could be better explained. Again I don't think you treat any entities what do you mean?
\\

{\em We agree, no treatments. Fixed the text.\\}


P19 L57 What learned theory are you referring to. Clarify.
\\

{\em We mean the Bayesian defect predictors. Fixed. \\}


P19 Figure 15 needs some more explanation.
\\

{\em We extended that discussion considering your comment below.\\}


P20 L 27 '.. the larger the variance .... ' you need to identify or speculate why this has happened.
\\

{\em We have now included text on the possible reasons. Fixed.\\}


P20 L34-35 This sentence is not in correct grammatical English!
\\

{\em 
This text has been added: 11We conjecture that this effect is due to the presence of occasional larger outliers in the training
data (the probability of discovering these increases as the training set grows inside and, if the learner
trains on them, then the resulting theory has more variance.)'' .\\}


P20 L41-50 Does not seem to me to explain why larger training sets result in larger variance give an example. This needs more thought and explanation.
\\

{\em see above comment.\\}


P20 L 55 This can' be skipped! Actually you have not addressed validity threats here - you just provide other researchers support for your results and give too much detail of their work L28 etc
\\

{\em We agree. Fixed\\}


P21 Again fig 17 is not a figure 
\\

{\em Fixed.\\}


P21 L37-L44 What are you saying: that after 3.7 months with one set of new modules in your company data that you can then use those to predict errors in any new systems to be developed? I am not sure that you can assume that there would be enough relevant data from that. You must convince us that this would be the case. It could help us to believe this if you had shown us the data you used in the CC NN and WC studies. 
\\

{\em In this experiment we report that 100 data samples are enough to build a WC defect predictor and using more data samples do not make the predictors any better. Cocomo model estimated reveal that 100 data samples can be implemented within 2 to 4 person months. We believe our point is clear here. Also the data used in the experiments are available online at PROMISE data repository as mentioned in the text and is explained in detail in data section.  \\}


You didn't actually wait 4 months then use the data and your model in the company's next system development, did you?  If so then say so.
\\

{\em We predict that this would be the case in general. For the extreme case of SOFTLAB we were able to collect data from previous projects within weeks and we describe this in the practical implications section .\\}


P22 L21-27 You need to list and describe the 19 common features and those that you used in WC Soft lab 
\\

{\em Please see new feature tables.\\}


P22 L41 -L42 Remove suggestive and generality and re-write.
\\

{\em Fixed.\\}


P23 L 57 But you can do this by examining all modules.
\\

{\em We agree. However, the purpose of defect prediction is to avoid examining all modules, since testing is considered to be the most expensive and time consuming phase of software development .\\}


P23 L61 I don't think you proved this hypothesis, firstly there was no well defined hypothesis and secondly there was no statistical testing done. Where was your hypothesis test and what was the p value?
\\

{\em We agree. We now define a formal hypothesis and report the significance level in NN section.\\}



P25 L7 What do you call strong evidence what were the results of any statistical test?
\\

{\em As all results provided in the paper, this result is also reported based on statistical significance/ validity. What we mean here is that observing the same results in the project data used in this paper, does not necessarily mean our results are externally valid for all projects. However, considering that we have used 10 projects from 8 different company is a strong indication that our results can be reproduced in any software project \\}


P25 L42 ... Than is perceived ."  I am not sure what the perception is or was. Give some reference to support the old perception about module error data (not effort estimation data) because you will be contradicting someone else's observations if you are correct. Or, you could drop the notion and state clearly how long it takes (or how much data we need to build a WC set). 
\\

{\em Agree. We have included Kitchenham et.al.'s comments on this issue. Their discussion is not limited on collecting effort data, rather they discuss the difficulties on general data collection process\\}


However, the readers need to be convinced about how few modules are needed in order to develop a WC database. For that they need to know more about the classifiers you used and how you developed your NB model. I still suspect that one sample system (albeit that there will be potentially many modules in a system) will not give enough information about all possible system modules that a company (that may well have many software developers and possible domains of development) might develop in the future. Readers need to be convinced. They need evidence. At the moment, I do think this paper manages that. 
\\

{\em Section 3.2 now contains 2 pages of notes on the details of building a NaiveBayes classifier.
Also, based on reviewer feedback, the entire structure of the paper has been rearranged. We beleive this
new draft is much clearer. \\}





\subsection{Replies to the Reviewer \#3 Comments}



Reviewer \#3: The manuscript has technical depth and demonstrates the author(s) considerable skill and experience with regards to certain types of machine learning algorithms. Nevertheless, the merit of the paper is somewhat limited and thus needs thorough re-work to become of significant interest for researchers and practitioners (instead of just being yet another exercise of finding a better prediction model for given data sets - in this case for predicting error-proneness of software modules).
\\

{\em We thank the reviewer for these comments. As we have explained in the Replies to the Editor Comments section, we have addressed all your concerns in this version of the manuscript. Below you may find our replies for the specific comments\\}

The weaknesses of the paper are listed below:

1. Originality:

Though it might be true that the authors are the first to apply k-Nearest Neighbor (k-NN) filtering in order to reduce false alarm rates for models derived from applying Bayes classifiers to inhomogeneous data sets (a typical property of for cross-company data sets), the authors are certainly not the first to observe that the quality of a prediction model can be influenced by the degree of similarity of the features in a data set to the features of the object of which a property is to be predicted. This is the guiding principle of all analogy-based prediction models (including approaches known as case-based reasoning (CBR) and optimized set-reduction (OSR)). It is surprising that analogy-based approaches are not discussed by the author(s) in the related work section.
\\

{\em We thank the reviewer for pointing out this issue. 
We have now added notes on reasoning by analogy methods in the literature review. 

The technical comment we would make though, is that to the best of our knowledge there are no publications in the defect prediction
literature  where the nearest neighbors are passed to a full-fledged classifier (in our case, Naive Bayes).
Further, our main  effect (that data from other sites can be quickly tuned to a local site using
automatic methods) has absolutely {\em not} been reported before.  \\}

2. External Validity:

External validity is somewhat limited. The results derived from 7
NASA data sets are - in their tendency - confirmed by a second
analysis conducted on 3 data sets from a very different organization
("SOFTLAB"). There is, however, no detailed discussion as to what
extent pd (probability of defect detection) and pf (probability of
false alarm) values transfer from one context to another. The
replication simply shows (as was to be expected) that the MEDIAN
value of both pd and pf decrease with the degree of homogeneity of
the underlying data set. \\

{\em 
We thank the reviewer for pointing out a discussion about the number of organizations considered. 
In the previous version we have discussed that NASA is an umbrella organization and the 7 projects used in this work came from different organizations. Therefore, we treat 7 NASA projects as if they were from 7 different companies. Considering the SOFTLAB projects, we analyze 8 different companies and a total of 10 projects. We have now included this justification in the text.
On the other issue (value transfer among contexts) we thank the reviewer, however we do not quite agree. We report median results and the statistical
tests that study {\em all} the values over all the distributions (Mann-Whitney U test). \\}

It is an open issue (not further discussed
by the authors) why in some cases k-NN filtered predictions are
better than predictions based on (emulated) within-company (WC)
data. Both are (rightly) considered as being more homogeneous than
the (emulated) cross-company (CC) data sets. In order to underpin
the theory about the relation between data homogeneity and prediction
quality, it might have been interesting to get a better understanding
of the differences between the cases in class "Observation1" (k-NN
filtered predictions on CC data produce higher pd and pf values
than unfiltered predictions on WC data) versus those in class
"Observation2" (k-NN filtered predictions on CC data produce lower
pd and pf values than unfiltered predictions on WC data).\\

{\em We thank the reviewer for this excellent comment. We have now included in the text, what the possible reasons may be (we can not be 100\% sure since we do not have access to the internals of these projects). On the other hand, we point out that
{\em no one} has a good predictor for what learners are best for what data sets.
We wish we had an answer to this issue, that would certainly be a brand new paper.
As it stands, we believe that our
result that alien data can be used locally after some NN pruning is worthy of publication.
This is the first such study and its results are very clear and those results hold for
systems as divergent as Turkish washing machines and NASA rockets.This is not a claim
of universal generality but it is founded on far more evidence that many other results
we could list. \\}


3. Impact/Relevance:

More important than the discussion of external validity is the practicality of the proposed approach. Looking at the detailed results of all experiments one can see that - for the same type of analysis! - pd and pf values vary greatly for different modules. For example, in experiment\#2 for NASA data (Fig. 11-14), median pd-values (pf-values) for WC-type predictions vary between 50 and 82 (21 and 36), for k-NN-typ predictions they vary between 56 and 82 (18 and 48), and for CC-type predictions they vary between 81 and 98 (27 and 94). For SOFTLAB data (Fig. 19-21), median pd-values (pf-values) for WC-type predictions vary between 40 and 100 (3 and 40), for k-NN-typ predictions they vary between 70 and 100 (25 and 38), and for CC-type predictions they vary between 88 and 100 (50 and 62).

For a practitioner it might make a considerable difference whether the pd-value in the context of a specific project is close to 40 or close to 100 (similarly for pf-values) in order to make appropriate decisions with regards to allocating people to quality assurance activities. There is no discussion whatsoever in the paper about the robustness of the prediction models generated by any of the presented methods. Thus, for a practitioner, the quality of the predictions are hard to judge in a specific situation. It would be desirable the author(s) discussed in some more depth the properties of the data sets in relation to the prediction quality in order to provide more guidance for the practitioner on when to use k-NN-type predictors versus WC-type predictors.
\\

{\em  
We thank the reviewer for this comment which produced  new Motivation and Practical Implications sections. Further, in section 8.3 we document what we know about current industrial practices
at finding defects- and our defect predictors do better than current industrial practice.
Defect prediction is not like formal verfication- it does not yield solid yes-no
answers. Rather, it is a statistical process that lets a manager select modules to be reviewed and
can bias test engineers towards the modules more likely to be defective. Yes, its results may vary
in the particular, but as a statistical process control methods, it yeilds results.
In section 2, and 8.3 we make the case that there exist communities of industrial practitioners
who find our work useful.  \\}


4. Presentation:

The paper is hard to read, particularly on pages 2-13. This is mainly due to the order in which the material is presented to the reader. The manuscript reads like a novel where the reader is given essential information in little bits scattered over the whole book. It is hard to make a concrete suggestion on how to re-organize the paper, but probably it would help if there was given a general introduction into the topic, then presented a research hypothesis, then the way how the hypothesis has been tackled (i.e., the experimental design), then the discussion of data sets and modelling techniques used, then the results, then the discussion of results (incl. discussion of threats to validity - not only external validity, btw.), then the conclusions (possibly involving some practical guidance for the practitioner).
\\

{\em All this issues have been resolved in this revised version. We have reorganized the manuscript as you suggested.  \\}


5. Other Major Issues:

Related to the issue of relevance/impact of the research presented in the manuscript is the question of what the words "within-company" and "cross-company" actually stand for. The author(s) themselve(s) make it clear that the term "company" does not really make sense in this context as NASA is certainly one "company" but obviously so heterogeneous that different sub-organizations within NASA may as well be considered separate "companies". In other words, the real question is not whether to use data "cross-company" or "within-company" but to assess the similarity of data repositories used, no matter whether they come from public external sources, from within the own company, from within the same product organization, from within the same development team, or from the very same developers. Thus, it would have made sense that the author(s) had considered more thoroughly related work in the field of machine learning that tries to define useful similarity measures to select data
sets that are most adequate for building prediction models. 
\\

{\em Thanks for pointing out this issue. We have put a lot of thought in this and could not quite come up with any better terms that would describe the work carried out. We most welcome any suggestions you may make, however in its current form (WC and CC) we are most satisfied, since these terms have also been used by Kitchenham et. al.. We agree that much future work could be devoted to (e.g.) our similarity
measures or the best value for ``k''. However, that work needs a baseline result against
which improvements could be assessed. We believe that this paper offers that baseline. \\}


Just as an aside: There also exits research (related to effort prediction) that claims there is no significant difference between WC-vs-CC data wrt. to prediction quality (--> Wieczorek/Ruhe). Another aside: There is no mentioning of the substantial body of work on defect prediction conducted by Briand et al.
\\

{\em At the suggestion of the other reviewers, we are de-emphasising the effort prediction work in this paper-
it only seemed to have confused the presentation.  Therefore,  while these researchers have done much work in empirical SE, we did not include them since their work is not directly relevant to the particular part of the field discussed here. We have now included an important work (we believe) by Arisholm and Briand for assessing the performance of defect predictors in practice.
And, in fact, during the course of this research we had several occasions to discuss with Lionel (Briand) at a personal level that there was no generality in theories learned from one company when applied to another. We believe in this paper we provide evidence of the opposite and  Lionel also agreed that "no generality" was an overstatement.   \\}


6. Other Minor Issues (listed in the order as they occurred when reading through the manuscript):

page 1 / lines 39-40: "Are the lessons learned from Project-X in Company-A relevant to Project-Y in Company-B? Theoretically, the answer is "no"." Why is the answer "no"? Why "theoretically"? What about the relevance of lessons learned from Project-X in Organization-A to Project-Y in organization A?
\\
page 2 / line 21: "While, WC data ..." --> "While WC data ..."
\\
page 3 / line 20: "Abrahansson" --> "Abrahamsson"
\\

{\em Above 3 issues are removed/ fixed in the new introduction.\\}

page 4 / Section B. Defect Prediction: --> the huge body of research on defect prediction using object-oriented code metrics is completely ignored. Why?
\\

{\em Our project features do not include any OO metric. We find it irrelevant to mention OO metrics.\\}


page 5 / Fig. 1: It is hard to compare the data presented in Fig. 1 with the data of experiments \#1, \#2, and \#3, as it is not clear whether it corresponds to the CC or WC setting; also, in Fig. 1, average values are used instead of medians. I suggest to provide one table that presents the results of experiments \#1 to \#3 in the same format as the data in Fig. 1 for better readability/comparability. Also, it would be nice to see some indication of the variance in pd and pf data shown in Fig. 1.
\\

{\em Thanks for your suggestion. We prepared such a table but the text become hard to organize. As it is, the text is near the relevant
figures now. With your suggestion, there would be much more "looking back" to the rest of the paper.
With your permission, we would like to keep it as it is in the current form.\\}

page 6 / line 20: Which experiment(s) support(s) the claim made in his line? For example, the median of the median-pd-values of the WC-case in experiment \#1 is 80, but the median of the median-pf-values is 31. Likewise, all median-pd-values of the CC-case in experiment \#1 are actually greater or equal 80, but again the median-pf-values are much larger than 26. Also the data from experiments \#2 and \#3 don't support the claim. So, where does it come from?

page 6 / lines 24-25: Do you mean (median-pd, median-pf) and (average-pd, average-pf) instead of (pd, pf)?
\\

{\em Fixed.\\}


page 6 / Fig. 2: It is not clear how the number of features shown in Fig. 2 map to the code-feature descriptions given in Fig. 3-5. When looking into the PROMISE data files one notices that the SOFTLAB data sets contain 30 descriptors, of which 29 are code-features and 1 is the categorization of whether a module is defective or not. This relationship is not well-described in the paper. 
\\

{\em New feature table fixes this issue.\\}

page 7 / line 45: What "theoretical evidence" do you have for the well-known, by many researchers observed (and intuitively clear) phenomenon that "The 'best' set of features may change from project to project"?
\\

{\em We referred to a prior publication and did not discuss further since we are not interested in feature selection. We removed this part.\\}

page 8 / lines 23-24: One "that" should be dropped.
\\

{\em Fixed.\\}

pages 9, 10, 12 / Fig. 3, 4, 5, 7: The content of these figures should be moved to appendices at the end of the paper.
\\

{\em Fig 3 is updated. 4-5 removed. Part of 7 removed or included in text.\\}

page 11 / Fig. 6: Please explain the values on the x- and y-axes. For example, what is the unit of the numbers shown on the x-axis? Possibly, this figure could be dropped as it doesn't provide much new information.
\\

{\em Fixed.\\}

page 12 / line 53: how exactly was the "median value around 8\%" computed. Based on all individual data samples, per project, ... ?
\\

{\em Median defect rate of 10 projects. Fixed:median=13\\}


page 12 / line 60: To improve understandability, please provide one example data set that exactly matches with one of the (aggregated) data sets in Fig. 8.
\\

{\em These are given in (old) Figure 10.\\}

page 13 / Fig. 8 and 9: Why sometimes "cm1", sometimes "CM1"? Why sometimes "pd", "pf", and sometimes "PD", "PF"?
\\

{\em Fixed.\\}

page 13: line 33: The "0\%" amd "100\%" markers are missing in Fig. 8.
\\

{\em Fixed.\\}


page 14 / Section C. Sanity Checks on Experiment \#1: This section does not "explore threats to the external validity of the Experiment\#1 conclusions". First of all, no conclusions have been presented (only results of various data analyses). Second, external validity deals with the generalisability of the results to other contexts (e.g., outside NASA).
\\

{\em Fixed.\\}


page 14 / line 58: How exactly is "lines of comments" defined? I.e., how is a line classified that has a mix of code and comment?
\\

{\em A mix line of code and comment is individually classified as 'lines of code and comment' .\\}


page 15: Fig. 10: What exactly was different in the analysis of the data sets as compared to results presented in Fig. 1 Was it only the learning algorithm (decision tree learner J48 versus Bayes classifier)? Similarly what was different in the analysis as compared to the one presented in paper [8]? Some modules analyzed in [8] are the same but the results are different.
\\

{\em Those works have different goals and follow different experimental procedures. For example in [8] feature subset selection was performed on another version of these data that includes 41 features. After publication of [8] some measurement issues arised with the datasets used and we didi not included them in this paper.\\}


pages 15-19 / Section V. Experiment\#2: NN-Filtered CC: Please summarize at one place what exactly was done differently between experiment\#1 and experiment\#2. It seems that in experiment\#1 different (sub-)sets of features were used. In experiment\#2 the sampling from the training set is repeated 100 times, how often did that happen in experiment\#1. In experiment\#1 there were 20 randomized orderings, was this also an issue in experiment\#2?
\\

{\em Fixed. Experiment designs are partially re-written.\\}


page 22 / line 21: Here you mention that the "SOFTLAB tables include 29 static code features", while Fig. 2 lists 30 "features". Implicitly, this hints to the fact that "features" (30) = "static code features" (29) + quality feature defective/non-defective (1). This is, however, not obvious from reading the manuscript and requires some effort on part of the reader to figure out.
\\

{\em Fixed.\\}


page 23 / Fig. 18 caption: Do you mean "Experiment \#1 results" or "Experiment\#3 results"? It's somewhat confusing that you label the data produced in Experiment\#3 as "Experiment\#1 results" just because you used the same algorithms than in Experiement\#1. Do you mean "Averaged" or should it read "Median"?
\\

{\em That is the experiment \# 1 replication results on SOFTLAB projects. By averaged, we meant using all SOFTLAB data not individual projects. Replaced averaged with overall.\\}


