Evaluation of text summarization systems is a thorny issue. Should the quality of system output be quantified through comparison with human summaries of the same texts? It is well-known that there are many good human summaries of the same document(s). How can we make the comparison if there is not a single gold standard? Should quality instead be measured by how well end users can carry out a task using summarization? Task-based evaluation can be time-consuming and it can be difficult to control experimental design so that the summary and not, for example, the interface or the difficulty of the document, impacts how well the end user does on the task.

In this talk, I will describe three different types of summarization evaluation we carried out at Columbia. Two are task-based, but involve different methods of measuring results. These evaluations were performed on a medical journal article summarization system and a news summarization system. The third evaluation used a new methodology called The Pyramid Method, developed at Columbia, to allow comparison of system summaries against multiple human gold standards. This method was used in NIST's DUC05, a forum for cross-site evaluation of summarization systems. In the talk, I will briefly describe, for each evaluation, the systems being evaluated as well as the evaluation methodology and results. There will be time for discussion of the many open-ended issues that remain for summarization evaluation.