Return to DUC Homepage
Document
Understanding
Conferences

Introduction
Publications
Data
Guidelines

Summary Quality Questions

 If the peer summary target size is 100 words, the evaluator reads the peer summary and then makes overall judgments as to the peer summary's quality, independent of the model. Questions 1-5 are within-sentence judgments; the rest are within- or across-sentence judgments. The answers are chosen in every case from the following set of 4 ordered categories: {0, 1-5, 6-10, more than 10}

  1. About how many gross capitalization errors are there?
    2. 	Examples: 
	- [new sentence] the new drugs proved beneficial.
	- PEOPLE WERE AMAZED WHEN THEY LEARNED THE DETAILS.
  2. About how many sentences have incorrect word order?
    3.    	
	Examples: 
   	- John before Mary the park visited
  3. About how many times does the subject fail to agree in number with the verb?
    4.    	Examples:
   	- The student see a teacher with a telescope
	- It was clear, that the reporters agrees with the idea.
  4. About how many of the sentences are missing important components (e.g. the subject, main verb, direct object, modifier) - causing the sentence to be ungrammatical, unclear, or misleading?
    5.    
   	Examples:
   	- The agreement, signed in London yesterday, .
   	- Mr. Smith to Washington where he met the senator.
   	- The exchange rate is %.
   	- Stewart, the builder.
  5. About many times are unrelated fragments joined into one sentence?
    6. 	Examples:
	- They run a refinery; two apples would be enough.
  6. About how many times are articles (a, an, the) missing or used incorrectly?
    7.   	Examples:
   	- Men saw woman with the telescope
   	- He picked up a book.  A book looked interesting.
   	- El Paso owns and operates refinery.
  7. About how many pronouns are there whose antecedents are incorrect, unclear, missing, or come only later?
    8.    	Examples:
   	- [opening sentence] Their agreement was signed in Oslo in 1933.
   	- Many Presidents were targets of assassins.  Pres. Reagan was
          wounded.  He was shot in the Ford's Theater in 1865.
  8. For about how many nouns is it impossible to determine clearly who or what they refer to?
    9.    	Examples:
	- The company agreed to negotiate.  [Which company?]
  9. About how times should a noun or noun phrase have been replaced with a pronoun?
    10.    	Examples: 
       	- Mr. John Smith went to DC.  Mr. John Smith saw the senator.
  10. About how many dangling conjunctions are there ("and", "however"...)?
    11.    	Examples: 
   	- [opening sentence]  However, they came to a good agreement.
  11. About many instances of unnecessarily repeated information are there?
    12. 	Examples: 

	- Yesterday's estimate does not include any projection for claims in
	Louisiana, which was also affected by the storm, although less
	severely than Florida.  But on the Florida losses alone, Hurricane
	Andrew becomes the most costly insured catastrophe in the US. 
	Louisiana was also affected by the storm. With Florida's Hurricane 
	Andrew losses added in, the total rises to Dollars 11.2bn. This
	does not include claims in Louisiana.
  12. About how many sentences strike you as being in the wrong place because they indicate a strange time sequence, suggest a wrong cause-effect relationship, or just don't fit in topically with neighboring sentences?

For data, past results, mailing list or other general information
contact: Lori Buckland (lori.buckland@nist.gov)
For other questions contact: Paul Over (over@nist.gov)
Last updated: Monday, 02-Dec-2002 15:49:15 UTC
Date created: Monday, 22-July-02