Group file operations for scalable tools and middleware

Group file operations are a new, intuitive idiom for tools and middleware - including parallel debuggers and runtimes, performance measurement and steering, and distributed resource management - that require scalable operations on large groups of distributed files. The idiom provides new semantics for using file groups in standard file operations to eliminate costly iteration. A filebased idiom promotes conciseness and portability, and eases adoption. With explicit semantics for aggregation of group results, the idiom addresses a key scalability barrier. We have designed TBON-FS, a new distributed file system that provides scalable group file operations by leveraging tree-based overlay networks (TBONs) for scalable communication and data aggregation. We integrated group file operations into several tools: parallel versions of common utilities including cp, grep, rsync, tail, and top, and the Ganglia Distributed Monitoring System. Our experience verifies the group file operation idiom i...

• Type: Journal
• Year: 2009
• Where: HIPC
• Authors: Michael J. Brim, Barton P. Miller
• Keywords: Distributed And Parallel Computing, File Operation, Group File Operation, HIPC 2009, IDIOM