A hit-miss model for duplicate detection in the WHO drug safety database

The WHO Collaborating Centre for International Drug Monitoring in Uppsala, Sweden, maintains and analyses the world's largest database of reports on suspected adverse drug reaction incidents that occur after drugs are introduced on the market. As in other post-marketing drug safety data sets, the presence of duplicate records is an important data quality problem and the detection of duplicates in the WHO drug safety database remains a formidable challenge, especially since the reports are anonymised before submitted to the database. However, to our knowledge no work has been published on methods for duplicate detection in postmarketing drug safety data. In this paper, we propose a method for probabilistic duplicate detection based on the hit-miss model for statistical record linkage described by Copas & Hilton. We present two new generalisations of the standard hit-miss model: a hit-miss mixture model for errors in numerical record fields and a new method to handle correlated...

• Type: Conference
• Year: 2005
• Where: KDD
• Authors: Andrew Bate, G. Niklas Norén, Roland Orre
• Keywords: Data Mining, Drug Safety Data, Drug Safety Database, KDD 2005, Post-marketing Drug Safety