Evaluating Smoothing Algorithms against Plausibility Judgements

Previous research has shown that the plausibility of an adjective-noun combination is correlated with its corpus co-occurrence frequency. In this paper, we estimate the co-occurrence frequencies of adjective-noun pairs that fail to occur in a 100 million word corpus using smoothing techniques and compare them to human plausibility ratings. Both class-based smoothing and distance-weighted averaging yield frequency estimates that are significant predictors of rated plausibility, which provides independent evidence for the validity of these smoothing techniques.

• Type: Conference
• Year: 2001
• Where: ACL
• Authors: Maria Lapata, Frank Keller, Scott McDonald
• Keywords: ACL 2001, ACL 2007, Corpus Co-occurrence Frequency, Human Plausibility Ratings, Smoothing Techniques