On the Statistical Consistency of DOP Estimators

A statistical estimator attempts to guess an unknown probability distribution by analyzing a sample from this distribution. One desirable property of an estimator is that its guess is increasingly likely to get arbitrarily close to the actual distribution as the sample size increases. This property is called consistency. Data Oriented Parsing (DOP) employs all fragments of the trees in a training treebank, including the full parse-trees themselves, as the rewrite rules of a probabilistic treesubstitution grammar. Since the most popular DOP-estimator (DOP1) was shown to be inconsistent, there is an outstanding theoretical question concerning the possibility of DOPestimators with reasonable statistical properties. This question constitutes the topic of the current paper. First, we show that, contrary to common wisdom, any unbiased estimator for DOP is futile because it will not generalize over the training treebank. Subsequently, we show that a consistent estimator that generalizes over...

• Type: Conference
• Year: 2003
• Where: CLIN
• Authors: Detlef Prescher, Remko Scha, Khalil Sima'an, Andreas Zollmann
• Keywords: CLIN 2003, Computational Linguistics, Estimator, Statistical Estimator Attempts, Training Treebank