HOME   CALL FOR TASKS CALL FOR INTEREST Workshop Submission Workshop PROGRAM NEWS   REGISTRATION TASKS (short list) TASKS (with rankings) DATA   [41 available]
LRE special issue  


Evaluation Exercises on Semantic Evaluation - ACL SigLex event



Available tasks:   1 

#12  Parser Training and Evaluation using Textual Entailment 

We propose a targeted textual entailment task designed to train and evaluate parsers. Recent approaches on cross-framework parser evaluation employ framework-independent representations such as GR and SD schemes. However, there is still arbitrariness in the definition of such a scheme and the conversion is problematic. Our approach takes this idea one step further. Correct parse decisions are captured by natural language sentences called textual entailments. Participants make a yes/no choice on a given entailment. It will be possible to automatically decide which entailments are implied based on the parser output only, i.e. there will be no need for lexical semantics, anaphora resolution etc.

- Final-hour trading accelerated to 108.1 million shares, a record for the Big Board.
  • 108.1 million shares was a record. – YES
  • Final-hour trading accelerated a record. – NO
The proposed task is desirable for several reasons. First, textual entailments focus on the semantically meaningful parser decisions. Trivial differences are abstracted away, which should result in a more accurate assessment of parser performance on real-word applications. Second, no formal training is required. Annotation will be easier and annotation errors will have a less detrimental effect on evaluation accuracy. Finally, entailments will be non-trivial since they will be collected by considering the differences between the outputs of different state-of-the-art parsers.
The participants will be provided with development (trial) and test sets of entailments and they will be evaluated using the standard tools and methodology of the RTE challenges. We hope the task will be interesting for participants with Parsing, Semantic Role Labeling, or RTE backgrounds.

Organizers: Deniz Yuret (Koc University)
Web Site: http://pete.yuret.com/

[ Ranking]

  • There will be no training data. The test data will be available on March 26
  • Closing competition : April 2nd

© 2008 FBK-irst  |  internal area