\documentclass[11pt]{article}

\setlength{\oddsidemargin}{0in}
\setlength{\evensidemargin}{0in}
\setlength{\topmargin}{-0.5in}
\setlength{\headheight}{0pt}
\setlength{\topskip}{0pt}
\setlength{\textwidth}{6.5in}
\setlength{\textheight}{9in}
\setlength{\parindent}{0pt}
\setlength{\parskip}{1mm}

\begin{document}

\begin{center} \bf \Large
Andrews, Menzies and Li \\
SBSE Special Issue Submission \\
--- \\
Authors' Responses to Reviewers
\end{center}

Thank you for your thoughtful and thorough reviews.  We have
revised the paper and give our responses to your comments
below.  Reviewers' comments are in {\tt teletype} font; our
responses are in Roman font.

There are two major areas of revision in this manuscript.
First, we have removed the section on the exploratory study,
pointing the reader instead at our ASE 2008 paper.  Second, we
have reworked the section on data mining of useful gene types,
so as to reflect the more systematic work we have done recently.
This work extends work
reported on in a paper in the PROMISE conference this year,
which we are submitting with this revised submission.


\section{Reviewer 1}

%\end{verbatim} \normalsize
%\small \begin{verbatim}

\small \begin{verbatim}
The paper describes a system which uses a genetic algorithm to
find a good setup for a randomised unit testing framework. 

One of the immediate concerns that come to mind is, how the
tuning of the GA compares to manually tuning the randomised
testing framework. The authors state that the randomised unit
testing framework requires expert knowledge of the framework in
order to make it effective. A GA typically also requires expert
knowledge, as there are so many parameters to tune, and by the
authors own admission, generally finding a good set of
parameters for a GA can be a bit of a "black art". The proposed
framework may be better suited for someone with good knowledge
of GAs, while the randomised unit testing framework without the
GA might be better suited for an expert in randomised unit
testing who has little or no knowledge of GAs. I feel this point
is not adequately addressed in the paper, and I would have liked
to see a study with independent experts, comparing RUTE-J and
Nighthawk.
\end{verbatim} \normalsize

Our intention is that the user can run Nighthawk by giving only
one parameter, a class to test (this being the most common kind
of unit to want to test).  The command-line parameters mentioned
in the ``top-level application'' section do not include anything
about the GA.  Essentially, we (the researchers) have done the
dirty work of finding good GA parameters; the user just has to
point the tool at a class.  This point was not brought out well
in the original submission, but is made more clearly in the
aforementioned section in this revision.

\begin{verbatim}
The authors imply that one of the benefits of their approach
over an evolutionary testing based approach is that their method
does not produce minimal test suites. While it is true that a GA
is typically aimed at different targets in turn, this does not
automatically result in a minimal test suite. Consider the
following example for illustration of a point:

foo(int b){
   int cnt = 0;
   while( 1){
	if(cnt >= b){
		//target 1
		goto while_break;
	}else{
		//target 2
	}
	cnt++;
   }
   while_break:
   if(cnt == 5)
	//target 3
  else
	//target 4
}

A minimal test suite to cover all targets would consist of 2
test cases. However, a "traditional" GA based strategy could
generate anything from 2-4 test cases, depending on order of
attempting targets, if you omit branches covered by previous
test cases (which you might not want to do),  etc. 
\end{verbatim} \normalsize

True; the reasoning is really comparing any tools which either
(a) generate a fixed set of test cases,
(b) generate many new test cases slowly, or
(c) generate many new test cases quickly.
Assuming that all the tools achieve the same
``thoroughness'' as measured by coverage, (c) is the best
choice, because coverage alone is not a reliable measure of
fault-finding effectiveness.  In the revised manuscript, we
present more evidence of this.

\small \begin{verbatim}
Further, on Page 10 the authors make the point that a minimal
coverage preserving test set has reduced fault detection
capabilities. Yet, as future work, they plan to include
"...coverage-preserving test case minimisation..." ?
\end{verbatim} \normalsize

Yes, if for instance you knew that test case K executed line L,
and you wanted to cut K down to the smallest sequence of calls
that will execute line L.  This could be used as an answer to
the question ``how does one execute line L?''.

\small \begin{verbatim}
Section C - Page 12:
I am not sure whether the authors are measuring efficiency or
effectiveness of weights, because they report on the amount of
code covered (effectiveness), not the time it took to achieve
more or equal coverage (efficiency)
\end{verbatim} \normalsize

This comment referred to the deleted exploratory study section.

\small \begin{verbatim}
Section 4 A - Page 15:
Figure 4, step 6 c); what is the rationale for always replacing
an existing value with a returned value for non-primitive types
and valid primitive types? E.g. why not use some probabilistic
replacement strategy?
\end{verbatim} \normalsize

We have inserted some text justifying this choice in the
section that describes the algorithm.

\small \begin{verbatim}
In Figure 3, step 2 b), if the method tryRunMethod returns a
failure or exception indication, the result is still placed in a
value pool? I think a little more detailed description of the
algorithm would help (maybe add a step 2 c) to Figure 3 )
\end{verbatim} \normalsize

Quite right -- we have clarified this point.

\small \begin{verbatim}
Page 19:
"The fitness function for a chromosome is calculated in a
manner..." . You calculate the fitness of a chromosome, not the
fitness function :)
\end{verbatim} \normalsize

That's a fair cop too -- we have rephrased this.

\section{Reviewer 2}

%\end{verbatim} \normalsize
%\small \begin{verbatim}

\small
\begin{verbatim}
This manuscript discusses the use of a genetic algorithm to
derive good parameters for the randomised (unit) testing.

The topic is relevant and interesting, and the paper is well
written and - on the whole - easy to read and understand.

Compared to the earlier conference paper by the same authors,
there is additional empirical work on optimising the GA using a
data mining approach, and an expanded section on related work.

However, the experimental approach is not sound from a
statistical point of view, and the material additional to the
authors' conference paper may not be sufficiently substantial to
distinguish this article from the conference paper.  It is
important to note that the additional material on optimisation
of the GA is not as clearly explained as the rest of the
manuscript.

Content
-------

General comments on content:

* (1) For what reason was a GA used as the 'upper-level' process
for finding optimal parameters for the randomised unit testing
'lower-level'.
\end{verbatim} \normalsize

The first paragraph in the ``System Description''/``GA'' section
gives our reasoning.

\small \begin{verbatim}
For example, could a Design-of-Experiments
methodology be used, or a deterministic technique, such as
Operational Research methods, be applied.  It is unclear why
there is a motivation to use a GA for this purpose as opposed to
other, potentially more effective and efficient, techniques.
\end{verbatim} \normalsize

Are you referring to (a) the idea of designing and running an
experiment with a classic design-of-experiments methodology,
or performing an OR analysis on the problem, or (b) tools that
simulate designing and running experiments or performing OR
analyses?

If (a), then yes, one could do an experiment or analysis to find
good randomized testing (RT) parameters for a particular unit, but without
any guarantees that this would extend to other units; one would have to do
such an experiment on every unit under test.  What we are aiming for is a
tool that can be run automatically, without any user intervention, to find
good RT
parameters.  Hence metaheuristic approaches were a natural choice.

If (b), then we are not aware of any such tools, but yes, these
would be good alternative approaches to compare a GA approach
to, along with simple hill climbing, simulated annealing, and
alternative GA formulations.  We would like to do such a
comparison in the future.  The focus of this paper, however, is
not the comparison of metaheuristic approaches, but the fact
that (at least) one such approach can be used to control a
randomized unit test generator so that it achieves high coverage.

As described in the Introduction, the main motivation of the work was not
to compare metaheuristic techniques, but to provide a way of getting high
coverage testing automatically.  In this regard, the main competing
technology is analysis-based approaches, which have complementary (but not
higher) strengths.  Hence we focused on comparison with that research and
on reporting what coverage statistics we were able to achieve.

The paper reports research on five dimensions of comparison:
\begin{enumerate}
\item Direct comparison to three other approaches, i.e.\
  Michael's GA-generating-test-cases approach, Visser et al.'s
  model-checking approach, and Pacheco et al.'s RANDOOP tool,
  each time using identical code to that of the other
  researchers;
\item Comparison across different classes in {\tt java.util};
\item Comparison of plain vs.\ enhanced test wrappers;
\item Comparison of deep vs.\ shallow target class analysis; and
\item Comparison of different sets of gene types.
\end{enumerate}
We would like to compare the GA approach to controlling RT with
other approaches to controlling RT, but since our focus here is
on comparing other automated unit testing approaches to RT with
{\it some} metaheuristic control, we have left that for future
work.  If some other control technique does even better than GA,
then so much the better.

\small \begin{verbatim}
* (2) * p11 line 40 (and also acknowledged, but not addressed,
in the Threats to Validity) Using these three classes taken from
java.util is very unlikely to form a representative sample: they
are likely to be coded to similar standards and have the similar
high-level purposes.  Including code from a variety of sources,
used in very different circumstances, not just code from a
single JDK, would be more convincing.  (Similarly for the java
1.5.0 collection and map classes, p23 Section VI, etc.)
\end{verbatim} \normalsize

The first comment above refers to the exploratory study, now
deleted.

The last, parenthetical comment refers to the main study,
retained in this manuscript.  We believe that there is no
essential disagreement here; our ``Threats to Validity'' section
reflects your concerns.  However, we also believe that the
{\tt java.util} Collection and Map classes are an extremely
important set of classes -- possibly the most heavily-used and
critical Java classes in use today.  One of the studies we
compare to (Visser et al.) reported coverage
results only for instrumented versions of a subset of the
methods in a subset of the Collection and Map classes.
Pacheco et al.\ ran their tool on the same software.
While Pacheco et al.\ also ran their tool on all the classes, they
are silent about what coverage they achieved on them.

Thus as far as we are aware, our paper is the first to report
actual results about coverage of all methods in all 16 actual
Collection and Map classes of {\tt java.util}, as achieved by a fully
automated tool.

However, recently we have applied Nighthawk to Apache Commons collections
classes,
which were built with a different development process and code style to
the Sun units.  Across the code of 35 units, we achieved an average of
86\% coverage, and the time taken was an average of 81.1824 seconds per
unit.  Should we report this case study and these numbers in more detail?
If so, we would appreciate advice about what to cut.

\small \begin{verbatim}
* (3) p12 line 14 (and elsewhere: p26, line 22; p32, line 23
etc.) Use of the t-test: the t-test assumes that the responses
are from a normal distribution (or in this context, that the
'stochastic noise') shows a normal distribution.  There is no
evidence presented that this is the case.  If the responses are
not from a normal distribution, such parametric tests can be
very unreliable.  A non-parametric test would be more suitable
in this case.
\end{verbatim} \normalsize

Some of the above applies only to the deleted exploratory study,
but some does apply to the rest.  You are quite right; we have
now applied the Shapiro-Wilk normality test where appropriate,
and a paired $t$ test and/or a paired Wilcoxon test
where appropriate,
and reported all the relevant data.

\small \begin{verbatim}
* (4) The comparison made in section VI between different
configurations of Nighthawk all used the same GA parameters (as
far as I can tell).  It's entirely possible that for different
settings of the GA, the comparison might have favoured different
configurations of Nighthawk, i.e. there might be interactions
between the GA setting and the best Nighthawk configuration.  A
more principled approach would be to 'tune' the GA for each of
the four configurations in an equivalent manner.  If the GA
parameters used where those that had been found (by trial and
error) to be best for one configuration (e.g. PN) in the
previous case study, then this might well have introduced a
significant bias in favour of PN.
\end{verbatim} \normalsize

Our opinion was that the more principled approach was rather to
fix one setting for the GA parameters, and compare Nighthawk
configurations using that same setting, to avoid the confounding
factor of GA parameter setting.  As far as we are aware (further
to the ``black art'' comment), it is difficult to tune a GA in a
systematic or principled way, so any attempt to do so would have
resulted in further confounding factors.

\small \begin{verbatim}
* (5) Compared to the rest of the paper, I found the section on
Data Mining-Based Optimization, particularly difficult to read and
understand, such as:
  - p28, line 16, the introduction of the terminology "sensors"
    and "actuators"
  - p28, paragraphs beginning at lines 25, 38 and 45
  - p27, line 49, the terminology "class" and "opposite class"
    (which are different from the usage of "class" as an OO in the
    rest of the manuscript)
  - the discussion in subsection B starting, particularly the
    paragraph starting at p30, line 26
\end{verbatim} \normalsize

We have made some changes to clarify this discussion.

\small \begin{verbatim}
(6) It is not explicitly stated that the technique is used for
testing object-oriented software, although such terminology is
used throughout.  It would help to clarify what aspect, if any,
are particular to OO software, and whether the technique is
compatible with the testing of non-OO code.
\end{verbatim} \normalsize

We have added some thoughts about this to the discussion about the tool.

\small \begin{verbatim}
p1, line 40 - index terms could include search-based
optimisation for software engineering or similar
\end{verbatim} \normalsize

Done.

\small \begin{verbatim}

p1, line 42 "Nighthawk uses only the Java reflection facility
... makings its general approach ... adaptable to other
languages."  It's not clear how the approach would be adaptable
if other languages did not provide an equivalent mechanism to
gather information about the SUT.
\end{verbatim} \normalsize

True -- the main point is that it collects no information about
the code within the methods, only about the classes, methods,
method
parameters and return values.  This has been clarified.

\small \begin{verbatim}
p4, line 20 "We compare Nighthawk to other systems ... showing
that it can achieve the same coverage levels." If this is the
case, what is the advantage in using Nighthawk.  Is it faster,
more efficient, more generally applicable, or does it in fact
achieve better coverage levels?
\end{verbatim} \normalsize

We have highlighted the advantages of randomized testing (fast
generation of many new test cases) better.

\small \begin{verbatim}
p6, section B - it's not clear how the discussion of analysis of
*white box, structural* test data generation techniques is
relevant to a manuscript on randomised unit testing.
\end{verbatim} \normalsize

This has been clarified.  (Our fitness function is based on coverage, so
Nighthawk is searching for chromosomes that improve coverage.)

\small \begin{verbatim}
* p9 Fig 1 (and 2?)  If x and y are integer values, then the
"spikes" at non-integer values are artifacts from arbitarily
joining the discrete points at integer values of x and y with an
arbitary 0 vertical axis value.  Instead of spike there are just
points at equal integer values of x and y.  (The argument still
holds - the points are a jump up from the plain - but the
diagram is misleading.   If x and y are real values, then there
are no spikes: there is only a ridge along the line x=y.
\end{verbatim} \normalsize

True -- our example was about {\it integer} values.  This has
been clarified, and a note added about what it would look like with
floating-point values.

\small \begin{verbatim}
p10 line 4 "We further consider not only numeric data, but data
of any type."  It isn't explained here, or later, how the
concept of "lo" and "hi" bound, or a "range" can be achieved for
non-ordinal data types.
\end{verbatim} \normalsize

This has been clarified.

\small \begin{verbatim}
p11, line 50 "We ran the two-level algorithm 50 times" (and
elsewhere, e.g., p21, line 23): why 30 times?  Why not 50, 100
or 10?  For example, was any analysis performed to show that 30
produced statistically robust results?
\end{verbatim} \normalsize

The exploratory study has been deleted.  ``10'' in the remaining
section was arbitrary.  Since it was impossible to predict the
variance in the results, it was not possible to calculate an
optimal sample size.  In this version, we have given means,
standard deviations, and 95\%
confidence intervals where appropriate.

\small \begin{verbatim}
p12, line 18 "statisticallly significant" - what significance
level was used?
\end{verbatim} \normalsize

95\%.  This has been made explicit everywhere it was absent.

\small \begin{verbatim}
p19, line 11 "Nighthawk uses ... settings ..."  Some
justification was made for these default values.  Was any work
done to show that the settings do indeed minimise the number of
chromosome evaluations as implied?
\end{verbatim} \normalsize

The goal was not actually to minimize the number of chromosome
evaluations, just to reduce them; the wording has been corrected.  The
population size and number of generations was reduced, which
clearly reduces the number of evaluations.  The mutation rate
was increased in order to increase diversity, to compensate for
the reduced population size and reduced number of generations.  This has
been clarified.

\small \begin{verbatim}
* p2 Fig 6 (and for all other results) - would be helpful to
give confidence intervals as well as mean values (where
appropriate) in order to give an indication of the variance in
the results
\end{verbatim} \normalsize

No means are reported in Figures 6 through 9  -- these are raw
numbers from one run each. However, there are some places
where means are reported, and 95\% confidence intervals have
been added there.

\small \begin{verbatim}
p22, line 38 "We than ran ..." It would be helpful to clarify
what hypothesis was being tested by the experiments described in
this paragraph, and what the results indicate.
\end{verbatim} \normalsize

This has been clarified.

\small \begin{verbatim}
p23, line 36 "... is a reasonable goal for system test ..." 
Although the intended meaning of "system test" used in this
quote isn't clear, it's possible that it is referring to a much
higher level test than the types of *unit* test performed in
this manuscript
\end{verbatim} \normalsize

True -- we have added more support for our claim.

\small \begin{verbatim}
p26, line 18 "... we report the number of seconds ... taken ...
to first achieve its best coverage."  While this may be
appropriate for experimental work, it is not suitable to compare
the practicality of the technique.  In practice, the user (and
the tool) will not know what the best coverage is, and so it
will be run for a set period of time - which may be much longer
than time at which the best coverage was found - and the best
coverage over the entire period.  Therefore, it would be
particularly useful to report confidence intervals (or variance)
in this case.
\end{verbatim} \normalsize

There are no means being reported here, so there is no
confidence interval to report -- these are raw numbers.
Actually, this methodology was taken from the work of Pacheco et
al., and we believe that it supports exactly the goals that you
support above:  users can use this data to inform them about how
long to let Nighthawk run.

\small \begin{verbatim}
p26, line 40, section VII - ideally, the Threats To Validity
section should come after the experimental work on feature
subset selection and encompass that work also.
\end{verbatim} \normalsize

This has been moved.

\small \begin{verbatim}
p31, line 31 - it would be helpful to include in the discussion
of the good performance of "upperBound", some speculation on why
"lowerBound" does not demonstrate a similar good performance.
\end{verbatim} \normalsize

This section has been reworked to reflect our more recent, more
systematic study, so this issue no longer arises.

\small \begin{verbatim}
Typographical and Formatting
----------------------------
p9 Fig 1: is the vertical axis value in this case a probability,
or just a fitness score?
\end{verbatim} \normalsize

This has been clarified.

\section{Reviewer 3}

%\end{verbatim} \normalsize
%\small \begin{verbatim}

\small \begin{verbatim}
My main reason for requesting a major revision of this article
is its similarity with the already published paper (ASE'07).
Although the authors take pains to point out the differences,
these only constitute a broader literature review (to be
expected, given the constraints imposed by the length of
conference papers) and a small section on feature subset and
optimization which does not, in my opinion, bring the paper up
to TSE standard. The other results in the paper are identical to
those reported in ASE and hence the majority of the paper has
nothing new to report. There was an opportunity here to expand
on the experiments and also address some of the threats to
validity identified by considering a wider set of target programs.
\end{verbatim} \normalsize

The ASE paper, reformatted in TSE format, took up 27 pages, and
the expanded paper took up 36 pages, so over 30\% new material
was added.  But these page counts do not fully characterize the large
delta between the ASE'07 paper and this TSE version:
\begin{itemize}
\item
In this manuscript, we have deleted the
exploratory study section from the ASE paper;
\item
All the FSS material in the second half of the paper is new. 
This FSS material not only is an extension of our report in the original TSE
draft but also of our FSS research conducted subsequently.
In our 
Promise09 paper, we made showed that FSS can lead to
reductions in the number of gene types.
Since then, we have conducted further research and our
results are startling. After refining the techniques of our PROMISE'09 paper,
we found yet another reduction method, this time leading to 10\% of the original
number of gene types. This paper reports the refined technique
developed after the PROMISE'09 paper (and this revised draft to TSE is
the first time this refined technique has been offered for
publication).
\end{itemize}
Also, if required,
we could also report in more detail about the
study on Apache Commons noted above, if you wish.  If so,
we would appreciate advice about what material to cut.

\small \begin{verbatim}
There are also a number of more minor issues which the authors
should consider if they choose to submit a revised version of
the paper:
- One of the main motivations behind the work is the avoidance
of any deep analysis of programs, although the drawbacks of thos
approaches which employ source/bytecode anaysis are not
identified.
\end{verbatim} \normalsize

We have expanded the discussion of those drawbacks in the first
two sections.

\small \begin{verbatim}
 Also it is pointed out that the use of a pool of
data can help solve some of the problems posed by equality
comparisons in predicates, but the construction of the pool, and
in particular the identification of the upper and lower bounds,
is not well explained. It is hard to imagine how these bounds
are identified in the general case.
\end{verbatim} \normalsize

The bounds (for numeric primitive types) are controlled by the
genes.  We have added a diagram which we hope clarifies this.

\small \begin{verbatim}
- Another element of confusion is the "weight" of the methods.
This is explained as its calling frequency but again there is no
explanation of how this is derived. This is disappointing as
this is shown to be an important factor in some case and again
it is hard to speculate whether this could be derived
automatically or whether it is important domain information
supplied by the user.
\end{verbatim} \normalsize

The relative weight of each method is controlled by a gene, as
explained in the figure containing the gene types.

\small \begin{verbatim}
- Small point at the foot of page 12 - a solution cannot be
"more optimal"
\end{verbatim} \normalsize

True enough!  Fixed.

\small \begin{verbatim}
-In the description of the Nighthawk system it is unclear why
the return values of the methods are important.
\end{verbatim} \normalsize

This has been clarified (see above in response to Reviewer 1).

\small \begin{verbatim}
-There is something strange about the classification of
constructors as initializers or reinitializers - it would appear
to be considered as both if it has no parameters. Perhaps this
is inteded...
\end{verbatim} \normalsize

You are correct -- this has been pointed out for clarity.

\small \begin{verbatim}
- Regarding the algorithm descriptions, an example would be
useful to illustrate the contents of concepts such as the value
pool (the constructionof which is also unclear) and show how the
algorithms operates (also a one-sentence description of the aim
of each algorithm would help).
\end{verbatim} \normalsize

We hope that the diagram has helped here.

\small \begin{verbatim}
- How is the success or failure of a test case determined?
\end{verbatim} \normalsize

The brief sentence about preconditions and result checking has
been expanded to clarify this.

\small \begin{verbatim}
- The meanings of the parameters p, g and m should be explained
for those readers not familiar with the GA literature.
\end{verbatim} \normalsize

The definitions are buried in the previous paragraph.  We have added a
sentence pointing out where they are, and have added discussion about why
we changed them.

\small \begin{verbatim}
- The paper talks about "the most fit chromosome" (section
IV-D). Is there always only one?
\end{verbatim} \normalsize

Not necessarily.  This is really ``the first chromosome
achieving the highest fitness recorded'' -- this has been clarified.

\small \begin{verbatim}
- In section V it would again be useful to know what lower and
upper bounds were empoyed on the data and how these were
identified, as i'm sure that these are crucial in understanding
and explaining the results.
\end{verbatim} \normalsize

These also are controlled by genes, as explained in the figure
containing the gene types.  We hope that the new diagram has clarified
this.

\small \begin{verbatim}
- In V-B please explain why Nighthawk was able to cover
significantly more lines of code when using the full target classes.
\end{verbatim} \normalsize

Done.

\small \begin{verbatim}
- The figure or 70-80% being a reasonable coverage goal quoted
at the end of section V-B is slightly misleading since it would
appear to relate to system test and these are unit tests that
are being carried out.
\end{verbatim} \normalsize

True -- we have added more support for our claim, as noted above.

\small \begin{verbatim}
- The last paragraph of section VI is confusing. It is stated
that t-test showed some results to be significantly different,
and in the following sentence it is claimed that "enriched
wrappers allowed Nighthawk to cover significantly more code
without running significantly longer". You can't have it both ways!
\end{verbatim} \normalsize

We have clarified this.  There was no statistically significant
difference in {\em time} between the (PN, EN) pair or between the
(PD, ED) pair, but there were statistically significant
differences in {\em coverage} between the (PN, EN) pair and between
the (PD, ED) pair.  Hence the conclusion is as we
stated it.

\section{Reviewer 4}

%\end{verbatim} \normalsize
%\small \begin{verbatim}

\small \begin{verbatim}
This paper presents Nighthawk, a two-stage system where a
high-level genetic algorithm is used to compute parameter values
used by a lower-level random test generation algorithm with the
goal of maximizing test coverage for unit testing. Results of
experiments with several examples of Java programs are discussed.

Overall, I found the paper interesting and mostly well written.

The key focus of the genetic-algorithm techniques used in this
work seems to be in deriving proper sequences and parameters of
method calls for unit testing of object-oriented software. It is
therefore closely related to Randoop [3]. Although the
heuristics used in [3] are not described using genetic-algorithm
terminology, I wonder how the two approaches really differ. A
more detailed comparison with [3] than the short one on page 6
would be welcome.
\end{verbatim} \normalsize

We currently devote two paragraphs (11 lines of text) to this work, and
make two points of contrast to our work (the breadth-first vs.\ 
depth-first nature of the algorithms, and the goals of the work).
As far as we can tell, Randoop does not use GA technology at all.
In addition, part of the present Comparison section directly
compares our results with theirs.  We
would be open to suggestions about what other specific points to discuss.

\small \begin{verbatim}
A key originality of Nighthawk seems to be its two-stage
algorithm, where genetic algorihms are not used to generate
tests directly, but rather set parameters in another random test
generation algorithm.  Although I am familiar with the general
principles of genetic algorithms, I am not an expert and I have
not kept up with the numerous applications of those algorithms
in software engineering, so I cannot comment on the novelty of
the approach. (It definitely sounds new to me.)
\end{verbatim} \normalsize

\small \begin{verbatim}
The comparison with analysis-based test generation approaches is
slightly biased. First, it is not true that (page 7) these
approaches "are as yet infeasible except for small software
units": for instance, see [Automated Whitebox Fuzz Testing,
NDSS'2008].
\end{verbatim} \normalsize

We can find no information about code size or code coverage in the NDSS
paper.  However, it is true that our claim was a judgement call.  We have
deleted it.

\small \begin{verbatim}
 Second, the argumentation made in section 2.D (page
8) about the complementarity of lighter-weight random testing
and heavier-weight analysis-based approaches makes sense only if
randomized testing remains lightweight.  For generating values
of inputs x and y satisfying "x==y", it seems cheaper to extract
this constraint and solve it directly once for all rather than
painfully converging (maybe -- no guarantees here) in a
convoluted way towards a solution through multiple iterations of
Nighthawk and many tests (see page 9).
\end{verbatim} \normalsize

You would think so, wouldn't you?  :-)  That's certainly the kind of thing
that we thought when we started research on randomized testing.  However,
the proof of the pudding is in the eating.  We have been consistently and
pleasantly surprised by the power of randomized unit testing, and we have
seen no coverage numbers better than what we can achieve.  We would like
to publish them in order to see if and how other groups can better them.

\small \begin{verbatim}
 To illustrate the
complementarity of both approaches, the authors could use an
example where the effectiveness of analysis-based methods is
questionable, such as inferring proper sequences of method calls
given only the APIs (type signatures) of those methods.
\end{verbatim} \normalsize

Is this the kind of example that you are looking for?:  We believe that
the main area in which randomized unit testing has an advantage over
analysis-based methods is in generating long sequences of calls that
create complex structures.  For instance, our RT 2006 paper reported on a
mutant of TreeMap in a deeply-nested block of rebalancing code, that took
a sequence of 14 ``put'' and ``delete'' calls to fail.  Such faults are
often out of reach of analysis-based approaches, since they cannot reach
the point in the state space where the failure occurs.
\vspace{1cm}

Thanks again to all the reviewers for the time and effort they put in to
reading and commenting on the paper.

%\small \begin{verbatim}
%Unfortunately, I cannot comment on the "data mining-based
%optimization" part (section 8) of the paper.
%\end{verbatim} \normalsize

\end{document}

