Reviews

Reviewer 1

Overall Rating

	6    (Accept: I would argue for accepting this paper.)

Expertise

	3    (Knowledgeable)

Contribution to HCI

The paper introduces a new type of interface, idea navigation, for information seeking and describes implementation and first evaluation of a system providing such an interface. Idea navigation is based on natural language processing and allows users to explore the content of a collection by narrowing their search based on sentence structure.

The Review

The paper describes an interesting new interface. The form of navigation is related to to faceted browsing but the way in which sentence structure is integrated is original and interesting.

The authors provide sufficient implementation detail so that the interface can be implemented by other groups. For interested readers it may be interesting to know which news corpus was used. If possible this should be included in the paper.

Evaluation could be more rigorous. Performance was not systematically compared to a baseline search interface. Such a comparison was only made in one case where users seemed to have problems solving the given task. Also, the authors do not explain how the tasks were devised, but claim that the system is "useful for solving realistic search tasks". However, I understand that the main focus was on presenting the new interface. For a full paper more rigorous evaluation would be needed, but I think this is fine for a CHI Notes paper.

Areas for Improvement

I found the paper well-written and well-organized.

Reviewer 2

Overall Rating

	6    (Accept: I would argue for accepting this paper.)

Expertise

	3    (Knowledgeable)

Contribution to HCI

The authors describe an extension to faceted navigation and search interfaces that they call "Idea Navigation". This interface extends the faceted navigation approach by allowing users to systematically filter targeted results based on subject-verb-object triples rather than on topics. In this paper, the authors describe this new type of navigation, and summarize the results of a user evaluation.

The Review

This paper presents a potentially valuable new approach to interactive navigation. The advantage of Idea Navigation over faceted navigation is that it allows users to more precisely filter targeted results.

Significance and benefit: The significance and benefit are in the design of the interface. Most interfaces that allow structured sentence-like queries (that I have seen) appear to be difficult to use. In contrast, this interface is an incremental extension to the normal faceted interface. As such, it has potential to be easy to use while still giving users fine-grained control over searching.

Validity: The authors' intentions in the evaluation did not appear to be precisely defined, and as such there is inadequate evidence presented either for the functionality of the interface or the usability of the interface. For example, the authors begin the Evaluation section by claiming that their formative evaluation was designed "To better understand the extent to which Idea Navigator is understandable and helpful...". In the very next paragraph they say that they "...were interested in testing the system's functionality (as opposed to the intuitiveness of the design)..." The evaluation tasks tested whether the users were able to understand and use the interface, rather than the system functionality (which could have been tested by professional testers--there's no need for a user study to test system functionality). The study proceeded to examine whether users could figure out how to use the system, as opposed to system functionality.

Giving the authors the benefit of the doubt and ignoring their claim that they were testing system functionality, the question remains what their evaluation was able to establish. They did seem to establish that users understood the new interface with instruction and demonstration ahead of time. I would have liked to have been able to find out whether users understood the new interface without instruction and demonstration.

There was also an implication in the Introduction that this design allows users to look for information that is "...more abstract or subjective". This claim was not supported in the evaluation. Tasks 2 and 3 asked users to find more subjective information. The authors do not report whether the users found subjective information in Task 2 (although they do mention that the users were able to use the interface to some extent).

 In Task 3, the users were unable to more easily find the subjective

information using the new interface. Furthermore, there is no reason to expect that the new interface would more easily allow searches for subjective information. Instead, an interface that allows searching over structured sentence-like information can only be expected to allow more precise searching: searching for information where the subject searched for really is the subject of the verb searched for an the object really is the object.

It would have been wonderful to see some metrics about the precision and recall of a search using the Idea Navigation interface versus a search using a standard faceted navigation and search interface (such as the one at http://flamenco.berkeley.edu/. Additionally, it would be wonderful to see a head-to-head user study evaluating the usability of standard search (like at www.google.com), faceted navigation and search, and Idea Navigation.

One final criticism: In the Implementation section, there was no information about the quality of the subject-verb-object extraction. Having incorrect extractions would have a negative effect on system functionality if that had really been tested. Further, it could also have a negative effect on the perceived usability of the system since it would create situations in which users are unable to retrieve targets even when using the correct filters.

Originality: As stated above, the new user interface described appears to be original.

Conclusion: Even with the criticisms given, I support the inclusion of this paper at CHI. I believe this is an original and potentially valuable interface design. This introduction could lead to more thorough evaluations and subsequent improvements to the design. The benefit of this interface approach is that it allows more precise searches. As such, there are certain to be industrial applications where any slight decrease in usability is made up for by increases in search precision.

Areas for Improvement

As mentioned above, the paper could have been improved with: - A careful definition of the objectives of the evaluation and a better evaluation design in order to measure against the objectives. - Omitting the implication that the new interface allows better searching for subjective information, since this is an odd expectation and was not supported by the results of the evaluation. - Some description of and metrics corresponding to the quality (precision and recall) of the sentence extraction mechanism.

Reviewer 3

Overall Rating

	2    (Reject: I would argue for rejecting this paper.)

Expertise

	3    (Knowledgeable)

Contribution to HCI

The system described in this paper uses a text summarization technique combined with an interface for browsing categories of search results, to help users find information in exploratory searches where they don't know what vocabulary to use.

The Review

The research problem addressed by this paper is that exploratory search is hard in systems that only support query-based searching (e.g., typing text into a search box like one does with Google or in an academic database like PsycINFO). The paper argues that one of he difficulties is discovering the right keywords to use when you don't know much about the domain in which you're searching.

The solution proposed by this paper is to use NLP (natural language processing) text summarization techniques to provide an interface that allows users to type a question as a query, and returns a list of categories related to that question. This allows users to use recognition (rather than recall) to select relevant categories by which to refine their search. The authors call this "faceted browsing".

I am a HCI researcher interested in NLP (natural language processing); I am familiar with text summarization as an approach but not well versed in the literature in the field. As such, I'm not sure after reading the paper whether the authors have developed an innovation for doing text summarization -- the "Implementation" section includes several references to what seem to be existing NLP technologies. Also, the idea of filtering information by category in a multiple categorization environment is not an innovation introduced by this paper. Perhaps their innovation is in combining existing techniques, or in the subject-verb-object approach. If so, this should be made more clear in the paper.

Regarding "faceted browsing": I have a somewhat different understanding of what it means for a system to support "faceted browsing". In Library and Information Science, a faceted classification system is one that allows items to be categorized along multiple dimensions (or facets), and categories are structured/controlled. iTunes has a faceted classification system; each song in iTunes has metadata associated with it in a number of different facets: song title, album title, track number, genre, etc. What separates a faceted classification system from other kinds of multiple categorization is the structure imposed by the facets.

The "structure" in the categories derived by the text summarization algorithm is dependent on the content and sentence structure in the news articles analyzed by the algorithm (if I understand the technique correctly). This means that the facets in this system are uncontrolled; they're based on whatever is in the news articles. In that sense, calling these categories "facets" might be misleading to people familiar with faceted classification as a technique that uses controlled vocabularies for information organization.

Regarding the user study: the paper also includes a brief summary of a user study in which 11 participants tried three different searches with the system. I found myself questioning the validity of the results reported, because the paper does not provide a description of how the data collected in the study were analyzed. The "Evaluation" section contains summaries describing how participants went about completing the search tasks; however, the paper does not provide quantitative measures of performance, nor does it include a description of the method they used to arrive at the qualitative descriptions. It is difficult to have confidence that the summaries describing user behavior are indeed representative without any information about how the analysis was conducted.

I feel like this paper does not make a strong contribution to the field of HCI; it is more appropriate for submission to a venue like SIGIR (http://www.sigir.org/) or WWW (http://www2008.org/) more focused on aspects of search and retrieval.

Areas for Improvement

None -- this paper was clearly written and very readable.