Evaluation Exercises on Semantic Evaluation - ACL SigLex event
#10 Linking Events and their Participants in Discourse
Semantic role labelling (SRL) has traditionally been viewed as a
sentence-internal problem. However, it is clear that there is an
interplay between local semantic argument structure and the
surrounding discourse. In this shared task, we would like to take SRL
of nominal and verbal predicates beyond the domain of isolated
sentences by linking local semantic argument structures to the wider
discourse context. In particular, we aim to find fillers for roles
which are left unfilled in the local context (null instantiations,
NIs). An example is given below, where the "charges" role ("arg2" in
PropBank) of cleared is left empty but can be linked to
murder in the previous sentence.
In a lengthy court case the defendant was tried for murder. In the
end, he was cleared.
There will be two tasks, which will be evaluated independently
(participants can choose to enter either or both):
For the Full Task the target predicates in the (test) data set
will be annotated with gold standard word senses (frames). The participants have to:
find the semantic arguments of the predicate (role recognition)
label them with the correct role (role labelling)
find links between null instantiations and the wider context
For the NIs only task, participants will be supplied with a
test set which is already annotated with gold standard local semantic
argument structure; only the referents for null
instantiations have to be found.
We will prepare new training and test data consisting of running text from the
fiction domain. The data sets will be freely available.
The training set for both tasks will be annotated with gold
standard semantic argument structure (see for example the FrameNet full text annotation) and linking information for null
instantiations. We aim to annotate the semantic argument structures
both in FrameNet and
style; participants can choose which one they prefer.
#11 Event Detection in Chinese News Sentences
Description The goal of the task is to detect and analyze some basic event contents in real world Chinese news texts. It consists of finding key verbs or verb phrases to describe these events in the Chinese sentences after word segmentation and part-of-speech tagging, selecting suitable situation description formula for them, and anchoring different situation arguments with suitable syntactic chunks in the sentence. Three main sub-tasks are as follows:
Target verb WSD: to recognize whether there are some key verbs or verb phrases to describe two focused event contents in the sentence, and select suitable situation description formula for these recognized key verbs (or verb phrases), from a situation network lexicon.
The input of the sub-task is a Chinese sentence annotated with correct word-segmentation and POS tags. Its output is the sense selection or disambiguation tags of the target verbs in the sentence.
Sentence SRL: to anchor different situation arguments with suitable syntactic chunks in the sentence, and annotate suitable syntactic constituent and functional tags for these arguments.
Its input is a Chinese sentence annotated with correct word-segmentation, POS tags and the sense tags of the target verbs in the sentence. Its output is the syntactic chunk recognition and situation argument anchoring results.
Event detection: to detect and analyze the special event content through the interaction of target verb WSD and sentence SRL.
Its input is a Chinese sentence annotated with correct word-segmentation and POS tags. Its output is a complete event description detected in the sentence (if it has a focused target verb).
The following is a detailed example to explain the above procedure:
For such a Chinese sentence after word-segmentation and POS tagging:
今天/n(Today) 我/r(I) 在/p(at) 书店/n(bookstore) 买/v(buy) 了/u(-ed) 三/m(three) 本/q 新/a(new) 书/n(book) 。/w (Today, I bought three new books at the bookstore.)
After the first processing stage: target verb WSD, we find there is a possession-transferring verb ‘买/v(buy)’ in the sentence and select the following situation description formula for it:
买/v(buy): DO(x, P(x,y)) CAUSE have(x,y) AND NOT have(z,y) [P=buy]
Then, we anchor four situation arguments with suitable syntactic chunks in the sentence and obtain the following sentence SRL result: