Highly scalable algorithm for distributed real-time text indexing

Stream computing research is moving from terascale to petascale levels. It aims to rapidly analyze data as it streams in from many sources and make decisions with high speed and accuracy in fields as diverse as security surveillance and financial services including stock trading. We specifically consider real-time text indexing and search with high input data rates (10 GB/s or more) along with small index ageoff(expiry) time. This makes it necessary to have maximal indexing rates for large volumes of data as well as minimal latency for indexing (time between start of indexing for a document and its availability for search) while maintaining very-low search response time. In addition, future massively parallel architectures with storage class memories will enable high speed in-memory real-time indexing, where index can be completely stored in a high capacity storage class memory. In this paper, we present the design of distributed datastructures and distributed real-time text indexing ...

• Type: Journal
• Year: 2009
• Where: HIPC
• Authors: Ankur Narang, Vikas Agarwal, Monu Kedia, Vijay K. Garg
• Keywords: Distributed And Parallel Computing, HIPC 2009, Indexing Algorithm, Real-time Text Indexing, Text Indexing