%% $Id: comments.txt,v 1.10 2006/02/27 19:43:25 gokhale Exp $ Reviewer #1 =========== Comment: -------- Summary: This paper describes how code optimization techniques such as constant propagation, layer-folding, memoization, code-refactoring, and aspect weaving can be applied to real-time middleware (in this case, TAO) to improve runtime performance (shown in a number of benchmarks). Discussion: I think, this paper is well-written and easy to read, and may be interesting to application developers using middleware as well as to middleware engineers. However, I am missing a strong technical contribution, a key idea, or at least some non-obvious results. I don't think manually applying established code optimization techniques to a software package that suffers by-design from substantial runtime overhead is an interesting research contribution by itself, in particular, when the hard part, i.e., automatic identification of too generic and thus inefficient yet optimizable code, is not addressed. Response: --------- While it is true that applying the specializations manually is not a win in the long run, our emphasis in this paper is to demonstrate that there remain numerous opportunities even within already highly optimized software frameworks that go unnoticed - or get masked - because of the excess generality and flexibility offered by these software frameworks. Our aim in this paper is to inspire the reader to take notice of the fact that for a family of related applications - called product lines - have very well scoped out functional and QoS requirements, because of which they do not need all the flexibility and generality of the middleware frameworks. We thus started with a hypothesis that even for highly optimized software frameworks, there possibly are opportunities to further optimize the system when unwanted generality from the middleware is removed and that such optimizations are not feasible without first removing the generality. This paper demonstrates this idea. One example of a somewhat specialized middleware framework is the Java micro edition (J2ME) which is a subset of the J2SE framework tailored for handhelds and other devices with memory constraints. The authors agree that the hard part is in automatically identifying the sources of these generalities. One approach to this is program analysis and compiler optimizations. However, our approach is not along these lines. Our approach hinges on the middleware developer identifying the sources of variability and generality in the source code by means of annotations. A specialization tool we have developed called FOCUS uses specialization metadata as input indicating which of the annotations in the middleware must be leveraged to perform specializations. The authors believe that a better approach to automation is to automatically synthesize the metadata input to the FOCUS tool. This work forms part of our current and future work. **Addressed in **: We added a brief explanation of the above in the Intro section under the bullet "Specializing Middleware for PLAs" Comment: -------- Details: Subcomment: ----------- - what is a product-line architecture? - what is a software family? Response to both questions: -------------------------- We use the term software family to mean products that are related to each other. A collection of such related products forms the product line. For example, Dell or Lenovo (previously IBM) have a product line of a particular desktop or laptop series e.g., Lenovo/IBM "T" series or "R" series. A product line is defined by a common scope comprising a number of commonalities including functional and QoS requirements. Each individual product is defined by the variability it demonstrates. **Addressed in **: In the paper we have defined this term and cited the SVC analysis paper by Weiss et. al in the first paragraph of the Intro section. Subcomment: ----------- - what do you mean by predictability? Response: --------- We use the term predictability for the real-time systems we are interested in. The goal is to ensure that the end to end latencies are bounded and have negligible jitter. **Addressed in **: Reworded appropriately in the Intro section. Subcomment: ----------- - figures 1, 2, 4, 5, 6, and 7 contain a lot of unexplained details. anything not explained might as well be removed. Response: --------- Most of these figures are middleware architectures. The necessary details from all these figures are covered in the paper. We have added a sentence or two in places where the explanation is not clear. However, the authors feel that explaining details of the figures will make the paper very dense and will affect the flow. For example, figure 4 shows the GIOP/IIOP packet header. This is akin to showing TCP/IP headers, where many papers in the literature do not explain every field. Our purpose in showing this header is to illustrate the important parts of the header that we have alluded to in the text, such as the byte order flag or the message type. **Addressed in **: Added in appropriate places where the figures are referenced. As stated above, too much detail was not necessary as explained above. Subcomment: ----------- - the related works section does not mention any related program analysis techniques. Response: --------- We have updated the related work section now based on comments from other reviewers. The list of changes are discussed below in Reviewer #4 and Reviewer #5's comments. **Addressed in **: See response to other reviewer comments. ============================================================================ Reviewer #2 ============ Comment: -------- The paper proposes the use of specialization techniques to address overheads due to middleware that is unnecessarily general for a family of applications. Motivated by on an application family used by Boeing, the authors identify a number of potentially unnecessary generalities in middleware systems, including the use of dynamic dispatching to call functions that turn out always to be the same and redundant creating and initialization of data structures. They then propose a number of specialization techniques, such as constant propagation and memoization to address these overheads, and propose a domain-specific language for describing where these specialization techniques should be applied. Finally, they show that their specialization techniques yield a significant performance improvement, particularly when all used together. Their approach also has little or no impact on the interface of the middleware, implying that no change is required to the applications. Overall, the paper represents a very interesting study of how techniques that have primarily been studied in terms of microbenchmarks and small examples scale up in terms of both usability and performance impact to the context of industrial strength code. The only weakness is that the paper is quite dense and long. Still, many researchers and students will learn from its content. Response: --------- Authors agree that the paper is dense. The goal of this work was to present a feasibility study of middleware specialization for product lines and to demonstrate a tool-driven approach. In order to showcase our work it was essential to demonstrate it in the context of a case study and also show the resulting improvements in observed QoS. This has led to an substantial increase in the length. **Addressed in **: Since this was a clarification question we have given our response above. Minor comments: --------------- Subcomment: ----------- Abstract: "(PLA)s" -> "(PLAs)" Response: --------- Fixed. Subcomment: ----------- Section 1: "[17,1]" -> "[1,17]" Response: --------- Fixed, and fixed several such other instances. Unfortunately, since this is handled by Bibtex and depends on the bib style used, different styles will give rise to different rendering. Subcomment: ----------- "that produce separate point solutions" What does this mean? Response: --------- We provided the meaning for this in the intro section. **Addressed in **: In the Intro section. Subcomment: ----------- Section 3.2: "To augment GNU autoconf" I found this a bit confusing, because autoconf is so far only associated with specializing platform generality (3.1.5). Response: --------- Yes. We have removed that from the paper. This was an error. Thank you for catching this! Subcomment: ----------- Section 3.2.2: "commenting regions of a program" Does this mean adding comments to the program or placing existing parts of the code within comments. If it is the latter, why do you also need the ability to remove code? Response: --------- Commenting regions requires annotations. That is, a start of a region to comment is identified by a marker and the end as well. Code removal uses regular expressions usually to modify or replace a single-line (in the current architecture). But, we agree that one could use code removal to do both. Subcomment: ----------- "During middleware evolution, such as feature addition/modification, middleware developers must respect the annotations" The next sentence doesn't seem like an example of respecting annotations. Response: --------- What we mean by respecting annotations is that, when something within a marker is replaced with something else. It should be still after the marker. If this is respected, that is the annotations are respected, then a transformation that uses the marker achieves the same effect even when the code has evolved. Subcomment: ----------- Section 3.3: All of your tables seem to appear in the middle of paragraphs. This is quite distracting, especially here where there is only one sentence of the paragraph before the table. Response: --------- Fixed wherever possible. In some cases Latex typesetting decides to move it to the top/bottom of the column and thereby break a paragraph. Subcomment: -------- Section 4.2.1: Listing 1: Does a substitution apply everywhere in the file? Is it possible to constrain it further? Response: -------- In the current design, substitution applies everywhere within a file. However one could possibly constrain it further using a within-start and within-end tags. That is substitute occurrences of a with that of b, but now constrain it further to a given region. This is a very good point. We will add this feature shortly. Subcomment: -------- Listing 2, 3: Is it correct that the return impl in Listing 2 has disappears in Listing 3? Response: --------- Yes it had. We have now fixed this. Thank you for catching this. Subcomment: -------- It would be better that the captions on the graphs correspond to the text used to refer to them. For example, the graphs say "max" but in Section 4.2.1, the text says "worst-case". Response: -------- We have changed worst-case measures to maximum measures in the section. Thanks for spotting this. Subcomment: ------- Section 4.2.3: "The (de)marshaling engine within TAO used several OS platform-specific capabilities that were automatically (un)set using GNU autoconf". It is not immediately clear whether this is describing your approach or something about the existing compilation of TAO that you want to undo. It would help to use "we" to make it clear what you do. Response: -------- We agree with what you have pointed out. What we did was to implement the specialization using two conditional compilation flag ACE_DISABLE_SWAP_ON_READ and CDR_IGNORE_ALIGNMENT. So an application developer who wants to leverage our specialization can turn on the specialization by enabling these flags via GNU's autoconf. As you have asked us to do so, we have made this clear in the text. Subcomment: ----------- Section 5.1: "For example, [18] have identified" It is not so nice to use a reference as a noun, especially here where it is somehow a plural noun, I guess because there is more than one author. Response: --------- Fixed. We used the authors name with et. al before the []. Subcomment: ----------- Section 5.2: "Technical challenges remaining include extending the automatic C program techniques in [14] to richer object-oriented languages, such as C++ and Java" Tempo has been extended to Java, although the applications on which the extension has been used are not the same as the ones described in [14]: Automatic Program Specialization for Java, Ulrik P. Schultz, Julia L. Lawall, and Charles Consel TOPLAS 25(4): 452-499 (2003). Also, the work about specialization has been described in detail in: Fast, Optimized Sun RPC using Automatic Program Specialization. In ICDCS'98, Amsterdam, May 1998. G. Muller, R. Marlet, E.N. Volanschi, C. Consel, C. Pu A. Goel. Finally, a language for decribing specialization opportunities in components has been described in: A Pragmatic Approach to Declaring Specialization Scenarios. Anne-Francoise Le Meur, Julia L. Lawall, and Charles Consel. Higher-Order and Symbolic Computation, 17(1), Mar 2004, pages 47-92. Response: --------- All three have been cited in Section 5.1 and not 5.2 since they are all partial evaluation related. Subcomment: ----------- Reference 24: There maybe a problem with the list of authors. Response: --------- We double checked it and it seems ok. Comment: -------- "Section 4.2 describes why we did not use FOCUS to resolve the (de)marshaling checks and deployment generality" Section 4.2 makes it clear that you did not, but it doesn't seem to explain why. Response: -------- The main reason we did not do is that the existing ACE CDR engine architecture used conditional compilation technique to leverage different OS primitives for buffer growth and copy operations. Therefore, to work with this existing framework, we did not use FOCUS. We don't think this information is important to the paper at all, therefore we are going to remove it. ============================================================================ Reviewer #3 =========== No comments to address from this reviewer. ============================================================================ Reviewer #4 ============ Comment: -------- Another source of related work seems to be compiler transformations for optimization. I was surprised not to see any references. Response: --------- The current structure of the related work section (Please see Reviewer #5 second comment) clearly illustrates the different types of specializations. In that partial evaluation, AOP to a certain extent, and empirically guided optimization techniques are all compiler assisted transformation techniques for program specialization. Comment: -------- The same confusion holds for Middleware annotation phase. What percentage of the source files are annotated? This implies how invasive will the annotations be during the specialization phase. Response: --------- This is a great question! To address this comment, we have added to each specialization the number of source files annotated to effect the specialization. In particular the metrics have been added to individual sections that discuss "Specialization automation". Comment: -------- In general the paper would be well served by extending the details of the FOCUS tool itself. Response: -------- In order to do this we have identified other comments that the reviewer has provided and systematically addressed them as shown below. We feel that addressing all these comments would help in clarifying the FOCUS tool as required by the reviewer. Sub-comment: ----------- What does it mean that "middleware developers capture code level transformations?" Are the middleware developers writing FSL descriptions saying what can be replaced with what? Response: --------- Yes! In order to make this clear Section 3.2.1 explicitly states that middleware developers use the FSL constructs to write the specialization transformations. Sub-comment: ----------- Does the FOCUS tool somewhat generate the FSL specifications? Response: --------- We realize what might have caused the confusion. The FOCUS introduction section 3.2.1 states that FOCUS is motivated by generative programming techniques and code-generative directives. Now we explicitly state that FSL does not auto generate the specializations. It is the middleware developers who manually have to write the different code transformations. Sub-comment: ------------ How are the specialization transformations verified as being appropriate as opposed to gratuitous? Response: --------- This is definitely a limitation as the tool itself cannot guarantee that the transformations are right. FOCUS is a vehicle to automate the specializations any other tool can be applied rather than FOCUS. We mention this in the Section that discusses the advantages and limitations of FOCUS. Sub-comment: ------------ How many specification files were generated for the specializations? Response: -------- To effect each specialization, we wrote one specialization file that captured all the transformations. Sub-Comment: ----------- How much expertise is needed by the PLA application developer to perform the specializations? Response: -------- A PLA application developer would need to understand the applicability and consequences of applying the different specializations. She would then choose the specializations that are applicable to her scenario at a "feature level" and the FOCUS tool would automate the specialization. In order to make this clear and explicit, the FOCUS discussion in particular Section 3.2.2 Phase 3 that explains the FOCUS specialization phase has been updated with discussion that make it explicit what a PLA developer needs to understand in order to automate the specializations. Sub-Comment: -------- This research was clearly done with both a middleware and PLA that the authors are intimately familiar with. Is this necessary for the techniques to be reapplied in a different scenario? Response: --------- The motivation sections discuss how the specializations are not tied to just the BasicSP scenario and are applicable to scenarios where the generality challenges exist. In order to emphasize this point, the Summary section 4.4 discuss how a PLA application developer would need to first see if the invariance assumptions that drive the specializations are applicable to the PLA she is developing. If so then, she need to understand consequences and applicability of the specializations. The paper provides these in the form of "Applicability and CORBA compliance" for each specializations. Sub-Comment: ----------- Is this reasonable in an environment where middleware is typically written in order to be widely applicable to different end-systems and the PLA developers do not have a tight connection with middleware developers. Response: -------- The section also mentions that some form of synergy is required by PLA application developer and the middleware developer for the above. The FOCUS tool does not currently solve this problem. We are however working on model level analysis tools to capture the invariance information at the model level. We have mentioned this there as a part of our on going research. ============================================================================ Reviewer #5 ============ Comment: -------- Is the implication that no one has ever used autoconf to customize software based on performance considerations? Response: --------- No. Autoconf is used as a vehicle to automate and deliver the specializations. Comment: -------- Specialization Techniques do not differ based upon the domain of application. In addition the reviewer states that prior work has shown specializations to be beneficial and therefore the discussion is irrelevant. Response: ----------- We agree with the reviewer that the specialization techniques themselves do not differ based on the domain of application. Therefore, we have re-structured the discussion as follows: we have changed the classification from a system application basis to focus on the underlying specialization. Therefore the subsections now are Partial evaluation technique, aspect oriented programming techniques, empirically guided optimizers and code synthesis techniques. However, we do not agree with the fact that the research initiatives are irrelevant. The related research illustrate systems research that have applied specialization such as operating systems. These serve as important related work to a reader of this paper also interested in knowing similar initiatives. Comment: -------- Some deeper discussion on the FOCUS tool Resolution: ----------- This was resolved as a part of Reviewer #4's comment on adding more details to FOCUS. Comment: ------- One system authors want to compare their work is 'C language. Response: --------- We have added the discussion on tick-C in our related work section under code synthesis techniques.