Employing Trainable String Similarity Metrics for Information Integration

The problem of identifying approximately duplicate objects in databases is an essential step for the information integration process. Most existing approaches have relied on generic or manually tuned distance metrics for estimating the similarity of potential duplicates. In this paper, we present a framework for improving duplicate detection using trainable measures of textual similarity. We propose to employ learnable text distance functions for each data field, and introduce an extended variant of learnable string edit distance based on an Expectation-Maximization (EM) training algorithm. Experimental results on a range of datasets show that this similarity metric is capable of adapting to the specific notions of similarity that are appropriate for different domains. Our overall system, MARLIN, utilizes support vector machines to combine multiple similarity metrics, which are shown to perform better than ensembles of decisions trees, which were employed for this task in previous w...

• Type: Conference
• Year: 2003
• Where: IJCAI
• Authors: Mikhail Bilenko, Raymond J. Mooney
• Keywords: IJCAI 2003, IJCAI 2007, Multiple Similarity Metrics, Similarity Metrics, Textual Similarity