Return to DUC Homepage
Document
Understanding
Conferences


Introduction
Publications
Data
Guidelines

DUC 2004: Call for Participation

Document Understanding Conference (DUC)

Conducted by:
National Institute of Standards and Technology (NIST)

With support from:
Defense Advanced Research Projects Agency (DARPA)




Over the last five or so years, we have witnessed a tremendous increase in interest in summarization research from both academia and industry. A DARPA program, Translingual Information Detection, Extraction, and Summarization (TIDES), specifically calls for major advances in summarization technology, both in English and from other languages to English (cross-language summarization). As part of this program, NIST began a new evaluation series in the area of text summarization, called the Document Understanding Conferences (DUC).

In DUC 2001 - 2003 a growing number of research groups (21 in 2003) have participated in the evaluation of generic and focused summaries of English newspaper and newswire data. Various target sizes (10 - 400 words) have been used and both single-document summaries and summaries of multiple documents (30 - 60 sets of around 10 documents per set) have been evaluated. Summaries have been manually judged on their content coverage and for their readability. Automatic evaluation has also been explored.

You are invited to participate in DUC 2004 (in Boston, first week in May 2004) and in the discussions concerning its definition, scheduling, etc. Preliminary information about DUC 2004 is available from the DUC website under Guidelines and then DUC 2004


http://duc.nist.gov/

Dissemination of DUC work and results other than in the (publicly available) conference proceedings is welcomed, but the conditions of participation preclude specific advertising claims based on DUC results. All summarization results submitted to DUC will be published in the Proceedings and archived on the DUC web site.

How to participate: Organizations wishing to participate in DUC 2004 should respond to this call for participation by submitting an application consisting of 1) contact information (organization name, full mailing address, voice and fax phone numbers, email of a main DUC contact) 2) a short paragraph on the organization's summarization approach, 3) an indication of whether this group has participated in DUC before. All applications should be submitted by 1. November 2003 to Lori.Buckland@nist.gov, but the sooner you apply, the sooner you can begin participating in the online discussions defining DUC 2004. Once you apply you will be subscribed to the DUC2004 email discussion list, which will be the means of communicating about DUC 2004.

Also, there will be 4 working groups beginning discussion right away. Email discussion lists have been created for each and are listed below along with a chairperson. When you apply for participation in DUC 2004, please indicate which group(s) you would like to actively participate in and what email address you will be using. Lori will then subscribe you.

The general goal for each will be to come up with some suggestions for improving the evaluation in the group's particular area. It will be up to the groups under the guidance of the chairs to figure out what they can likely accomplish in what time frame and come up fairly quickly with plan to get that done.

  • duc.newgenre - Getting on to summarizing material in a genre other than newswire/paper (Simone Teufel)
  • duc.qualquests - Reviewing and revising the linguistic quality questions in the light of changing tasks and lessons learned (Ani Nenkova)
  • duc.agreement - Issues associated with nature of (dis)agreement among human summarizers, how to reduce/tolerate/exploit it. (Hans van Halteren)
  • duc.autoeval - Automatic evaluation methods - what can('t) they tell us, how are they best used, ... (Chin-Yew Lin)

Any questions concerning DUC should be sent to Paul.Over@nist.gov. Late applications may be accepted if resources allow, but in no case will sample or test data be released to groups who have not applied.

For data, past results, mailing list or other general information
contact: Lori Buckland (lori.buckland@nist.gov)

For other questions contact: Paul Over (over@nist.gov)
Last updated: Wednesday, 23-Jul-2003 08:33:03 EDT
Date created: Friday, 13-September-02