Repositories like arXiv1 and knowledge bases like CiteSeer2 are increasingly becoming central to academicians and researchers. However, current systems provide too little semantic support like determining the topic of a paper, modelling the interests of a user, etc. Silverfish is a contextual knowledge extraction and aggregation system that adds a significant amount of semantics to academic repositories. Silverfish automatically extracts pertinent key phrases and models semantic content in the form of a back end cooccurrence graph. This is used for "intelligent" operations like recommendations, finding related content and semantics based routing of announcements.