Adjudicator Agreement and System Rankings for Person Name Search

We have analyzed system rankings for person name search algorithms using a data set for which several versions of ground truth were developed by employing different means of resolving adjudicator conflicts. Thirteen algorithms were ranked by F-score, using bootstrap resampling for significance testing, on a dataset containing 70,000 romanized names from various cultures. We found some disagreement among the four adjudicators, with kappa ranging from 0.57 to 0.78. Truth sets based on a single adjudicator, and on the intersection or union of positive adjudications produced sizeable variability in scoring sensitivity

• Type: Conference
• Year: 2008
• Where: LREC
• Authors: Mark Arehart, Chris Wolf, Keith J. Miller
• Keywords: Adjudicator, Adjudicator Conflicts, Education, LREC 2008, Truth Sets