Towards a More Careful Evaluation of Broad Coverage Parsing Systems

Since treebanks have become available to researchers a wide variety of techniques has been used to make broad coverage parsing systems. This makes quantitative evaluation very important, but the current evaluation methods have a number of drawbacks such as arbitrary choices in the treebank and the difficulty in measuring statistical significance. We suggest a more detailed method for testing a parsing system using constituent boundaries, with a number of measures that give more information than current measures, and evaluate the quality of the test. We also show that statistical significance cannot be calculated in a straightforward way, and suggest a calculation method for the case of Bracket Recall.

• Type: Conference
• Year: 1996
• Where: COLING
• Authors: Wide R. Hogenhout, Yuji Matsumoto
• Keywords: COLING 1996, COLING 2008, Coverage Parsing Systems, Current Evaluation Methods, Statistical Significance