Storage systems in emerging large-scale (a.k.a. peta-scale) computing systems often introduce a performance or scalability bottleneck. To deal with these limitations we propose a new operational approach: versatile storage, an application-optimized and highly configurable storage system that harnesses node-local resources, is configured and deployed at application deployment time, and has a lifetime dependent on the application lifetime. Our prototype evaluation, using synthetic and application-level benchmarks, on a small cluster as well as on a 96K processor machine, provides evidence that the versatile storage approach can bring valuable benefits to large scale deployments in terms of storage system performance and scalability. Categories and Subject Descriptors D.4.3 (Operating Systems): File Systems Management Distributed file systems. D.4.7 (Operating Systems): Organization and Design - Distributed systems. C.4 (Performance of Systems) Design studies. General Terms Performance, ...