Scalable Distributed Reasoning Using MapReduce

We address the problem of scalable distributed reasoning, proposing a technique for materialising the closure of an RDF graph based on MapReduce. We have implemented our approach on top of Hadoop and deployed it on a compute cluster of up to 64 commodity machines. We show that a naive implementation on top of MapReduce is straightforward but performs badly and we present several non-trivial optimisations. Our algorithm is scalable and allows us to compute the RDFS closure of 865M triples from the Web (producing 30B triples) in less than two hours, faster than any other published approach.

• Type: Conference
• Year: 2009
• Where: SEMWEB
• Authors: Jacopo Urbani, Spyros Kotoulas, Eyal Oren, Frank van Harmelen
• Keywords: Compute Cluster, Internet Technology, RDF Graph, Scalable Distributed Reasoning, SEMWEB 2009