Data redundancy is the most effective means to ensure storage reliability and data availability. Traditional redundancy mechanisms mainly include replication and erasure code (EC).
Do you remember how to treat your graduation thesis? Keep a copy on your computer, another copy on a USB drive, and yet another copy on a cloud storage service. You might even store multiple copies on different cloud storage services. You do this with trepidation, fearing that months of hard work (not really...) might go to waste. This is replication. The multiple copies strategy involves storing data as multiple replicas (usually 3 replicas). When one replica is lost, it can be recovered from the other replicas. Therefore, the disk utilization of 3 replicas is 1/3, allowing for the loss of 2/3 of the replicas.
Clearly, the disk utilization of the replication mode is not high. To improve this, we introduce the mechanism of erasure code, which can solve the problem of disk utilization. The operation can be roughly divided into 3 steps:
- Divide a file into K data blocks.
- Connect these K data blocks through a certain method (such as linear relationship or matrix relationship) to generate M parity blocks.
- When some data blocks are lost, use the parity blocks to recalculate the lost data blocks.
In this case, the disk utilization is: K/(K+M), and the allowed number of lost replicas is: M/(K+M). The principle of data recovery mainly utilizes matrix calculations, with the key term being Vandermonde matrix. For more details on the recovery principle, please refer to the references at the end of the document.
AnyShare Enterprise/Express7 by Aishu supports both replication mode and EC mode, with specific features including:
- 1-node and 2-node configurations adopt the 3-replica mode, while configurations with 3 or more nodes can be upgraded to support EC mode. Therefore, when there are fewer nodes, it is sufficient to consider the 3-replica mode.
- In the scenario of capacity expansion and upgrade, the object storage pool supports a mixed mode of 3 replicas and EC.
- AnyShare supports various EC strategies such as 4+2:1, 4+2, 6+3, etc. Among them, 4+2:1 refers to each data stripe consisting of 4 data blocks and 2 coding blocks, allowing for the failure of any 2 disks or any 1 node without affecting data integrity. This strategy requires a minimum of 3 nodes. On the other hand, 4+2 refers to each data stripe consisting of 4 data blocks and 2 coding blocks, allowing for the failure of any 2 disks or any 2 nodes without affecting data integrity. This strategy requires a minimum of 6 nodes.
- In general, EC mode has higher performance, disk utilization, and security than the 3-replica mode. However, the security of 4+2:1 is not as good as that of 3 replicas.
References:
freshyuan, "An Exploration of EC in Ceph Distributed Storage"