HOME   CALL FOR TASKS CALL FOR INTEREST Workshop Submission Workshop PROGRAM NEWS   REGISTRATION TASKS (short list) TASKS (with rankings) DATA   [41 available]
LRE special issue  


Evaluation Exercises on Semantic Evaluation - ACL SigLex event



Available tasks:   1 

#10  Linking Events and their Participants in Discourse 


Semantic role labelling (SRL) has traditionally been viewed as a sentence-internal problem. However, it is clear that there is an interplay between local semantic argument structure and the surrounding discourse. In this shared task, we would like to take SRL of nominal and verbal predicates beyond the domain of isolated sentences by linking local semantic argument structures to the wider discourse context. In particular, we aim to find fillers for roles which are left unfilled in the local context (null instantiations, NIs). An example is given below, where the "charges" role ("arg2" in PropBank) of cleared is left empty but can be linked to murder in the previous sentence.

In a lengthy court case the defendant was tried for murder. In the end, he was cleared.


There will be two tasks, which will be evaluated independently (participants can choose to enter either or both):
For the Full Task the target predicates in the (test) data set will be annotated with gold standard word senses (frames). The participants have to:

  • find the semantic arguments of the predicate (role recognition)
  • label them with the correct role (role labelling)
  • find links between null instantiations and the wider context (NI linking)

For the NIs only task, participants will be supplied with a test set which is already annotated with gold standard local semantic argument structure; only the referents for null instantiations have to be found.


We will prepare new training and test data consisting of running text from the fiction domain. The data sets will be freely available. The training set for both tasks will be annotated with gold standard semantic argument structure (see for example the FrameNet full text annotation) and linking information for null instantiations. We aim to annotate the semantic argument structures both in FrameNet and PropBank style; participants can choose which one they prefer.

Organizers: Josef Ruppenhofer (Saarland University), Caroline Sporleder (Saarland University), Roser Morante (University of Antwerp), Collin Baker (ICSI, Berkeley), Martha Palmer (University of Colorado, Boulder)
Web Site: http://www.coli.uni-saarland.de/projects/semeval2010_FG/

[ Ranking]

  • Test data release: March 26th
  • Closing competition : April 2nd

© 2008 FBK-irst  |  internal area