Evaluation Exercises on Semantic Evaluation - ACL SigLex event
#10 Linking Events and their Participants in Discourse
Semantic role labelling (SRL) has traditionally been viewed as a
sentence-internal problem. However, it is clear that there is an
interplay between local semantic argument structure and the
surrounding discourse. In this shared task, we would like to take SRL
of nominal and verbal predicates beyond the domain of isolated
sentences by linking local semantic argument structures to the wider
discourse context. In particular, we aim to find fillers for roles
which are left unfilled in the local context (null instantiations,
NIs). An example is given below, where the "charges" role ("arg2" in
PropBank) of cleared is left empty but can be linked to
murder in the previous sentence.
In a lengthy court case the defendant was tried for murder. In the
end, he was cleared.
There will be two tasks, which will be evaluated independently
(participants can choose to enter either or both):
For the Full Task the target predicates in the (test) data set
will be annotated with gold standard word senses (frames). The participants have to:
find the semantic arguments of the predicate (role recognition)
label them with the correct role (role labelling)
find links between null instantiations and the wider context
For the NIs only task, participants will be supplied with a
test set which is already annotated with gold standard local semantic
argument structure; only the referents for null
instantiations have to be found.
We will prepare new training and test data consisting of running text from the
fiction domain. The data sets will be freely available.
The training set for both tasks will be annotated with gold
standard semantic argument structure (see for example the FrameNet full text annotation) and linking information for null
instantiations. We aim to annotate the semantic argument structures
both in FrameNet and
style; participants can choose which one they prefer.