DUC 2003: Call for Participation

Over the last five years, we have witnessed a tremendous increase in interest in summarization research from both academia and industry. A DARPA program, Translingual Information Detection, Extraction, and Summarization (TIDES), specifically calls for major advances in summarization technology, both in English and from other languages to English (cross-language summarization). As part of this program, NIST began a new evaluation series in the area of text summarization, called the Document Understanding Conferences (DUC).

The first of these evaluations, DUC 2001, took place last September in New Orleans. Fifteen groups participated in the evaluation of generic summaries of newspaper and newswire data. Various levels of text compression were tested, with both single summaries and summaries of multiple documents (30 sets of around 10 documents per set) being evaluated. Summaries were examined for their coverage and for their readability by 10 human judges using software developed at USC/ISI.

In DUC 2002, 17 research groups participated with systems that automatically created summaries of document sets at four compression levels and for each document within the set. The summaries were again judged by humans for content coverage and readability against manually created summaries. Manual extracts were also created by NIST and served as the basis for automatic evaluation of automatically created abstracts.

You are invited to participate in DUC 2003. Some sample material will be available from NIST, and test material will be distributed the end of January 2003. NIST will collect and evaluate the results in the second half of February using the same group of people that created the reference summaries. The DUC 2003 workshop is planned for late May - we hope in conjunction with the HLT/NAACL conference in Edmonton, Canada.

Dissemination of DUC work and results other than in the (publicly available) conference proceedings is welcomed, but the conditions of participation preclude specific advertising claims based on DUC results. All summarization results submitted to DUC will be published in the Proceedings and archived on the DUC web site.

More information on DUC 2003, including the schedule, the data, and the plans for evaluation will be available on the DUC web site as preparations are completed. Information on previous DUC workshops is also available there:

How to participate: Organizations wishing to participate in DUC 2003 should respond to this call for participation by submitting an application consisting of 1) contact information (organization name, full mailing address, voice and fax phone numbers, email of a main DUC contact) and 2) a short paragraph on the organization's summarization approach. All applications should be submitted by 1. November 2002 to Lori.Buckland@nist.gov.

Any questions concerning DUC should be sent to Paul.Over@nist.gov. Late applications may be accepted if resources allow, but in no case will sample or test data be released to groups who have not applied.