Content-based document routing and index partitioning for scalable similarity-based searches in a large corpus