R-ADMAD: high reliability provision for large-scale de-duplication archival storage systems

Data de-duplication has become a commodity component in dataintensive systems and it is required that these systems provide high reliability comparable to others. Unfortunately, by storing duplicate data chunks just once, de-duped system improves storage utilization at cost of error resilience or reliability. In this paper, R-ADMAD, a high reliability provision mechanism is proposed. It packs variablelength data chunks into fixed sized objects, and exploits ECC codes to encode the objects and distributes them among the storage nodes in a redundancy group, which is dynamically generated according to current status and actual failure domains. Upon failures, R-ADMAD proposes a distributed and dynamic recovery process. Experimental results show that R-ADMAD can provide the same storage utilization as RAID-like schemes, but comparable reliability to replication based schemes with much more redundancy. The average recovery time of R-ADMAD based configurations is about 2-6 times less than RA...

• Type: Journal
• Year: 2009
• Where: ICS
• Authors: Chuanyi Liu, Yu Gu, Linchun Sun, Bin Yan, Dongsheng Wang
• Keywords: Data De-duplication, Distributed And Parallel Computing, General Terms Reliability, ICS 2009, Reliability Provision Mechanism