The linguistic quality questions are targeted to assess how readable and fluent the summaries are, and they measure qualities of the summary that DO NOT involve comparison with a model summary or DUC topic. The information content and responsiveness of the summary are measured separately in the "responsiveness" part of the evaluation. All linguistic quality questions require a certain readability property to be assessed on a five-point scale from "1" to "5", where "5" indicates that the summary is good with the respect to the quality under question, "1" indicates that the summary is bad with respect to the quality stated in the question, and "2" to "4" show the gradation in between. For each question, please try to assess the quality of the summary only with respect to the property that is described in the question. 1 Grammaticality The summary should have no datelines, system-internal formatting, capitalization errors or obviously ungrammatical sentences (e.g., fragments, missing components) that make the text difficult to read. 1. Very Poor 2. Poor 3. Barely Acceptable 4. Good 5. Very Good 2 Non-redundancy There should be no unnecessary repetition in the summary. Unnecessary repetition might take the form of whole sentences that are repeated, or repeated facts, or the repeated use of a noun or noun phrase (e.g., "Bill Clinton") when a pronoun ("he") would suffice. 1. Very Poor 2. Poor 3. Barely Acceptable 4. Good 5. Very Good 3 Referential clarity It should be easy to identify who or what the pronouns and noun phrases in the summary are referring to. If a person or other entity is mentioned, it should be clear what their role in the story is. So, a reference would be unclear if an entity is referenced but its identity or relation to the story remains unclear. 1. Very Poor 2. Poor 3. Barely Acceptable 4. Good 5. Very Good 4 Focus The summary should have a focus; sentences should only contain information that is related to the rest of the summary. 1. Very Poor 2. Poor 3. Barely Acceptable 4. Good 5. Very Good 5 Structure and Coherence The summary should be well-structured and well-organized. The summary should not just be a heap of related information, but should build from sentence to sentence to a coherent body of information about a topic. 1. Very Poor 2. Poor 3. Barely Acceptable 4. Good 5. Very Good