Adaptive-and-Resolvable Fractional Repetition Codes Based on Hypergraph

Abstract

Fractional repetition (FR) codes can achieve exact uncoded repair for multiple failed nodes, with lower computational complexity and bandwidth overhead, and effectively improve repair performance in distributed storage systems (DSS). The actual distributed storage system is dynamic, that is, the parameters such as node storage overhead and number of storage nodes will change randomly and dynamically. Considering that traditional FR codes cannot be flexibly applied to dynamic distributed storage systems, a new construction scheme of adaptive-and-resolvable FR codes based on hypergraph coloring is proposed in this paper. Specifically, the linear uniform regular hypergraph can be constructed based on the heuristic algorithm of hypergraph coloring proposed in this paper. Then edges and vertices in hypergraph correspond to nodes and coded packets of FR codes respectively, further, FR codes is constructed. According to hypergraph coloring, the FR codes can achieve rapid repair for multiple failed nodes. Further, FR codes based on hypergraph coloring can be generalized to heterogeneous distributed storage systems. Compared with Reed-Solomon (RS) codes, simple regenerating codes (SRC) and locally repairable codes (LRC), adaptive-and-resolvable FR codes have significant advantages over repair locality, repair bandwidth overhead, computational complexity and time overhead during repairing failed nodes.

1. Introduction

Nowadays, the information data is growing explosively. Due to significant characteristics of high throughput, high availability and high scalability, the large-scale distributed storage systems have been widely deployed to store massive data. However, node failures are common in large-scale distributed storage systems. Thus some redundancy strategies are usually adopted. The traditional redundancy strategies are replication mechanism [1] and erasure codes strategy [2]. The replication mechanism is to store multiple replicas of original file in different nodes, so storage overhead is too high. The erasure codes strategy requires high network bandwidth during repairing failed nodes, so repair bandwidth overhead is too high.

Dimakis et al. proposed the regenerating codes (RC), which can effectively reduce node storage overhead and repair bandwidth overhead [3]. Then Rashmi et al. proposed two classes of regenerating codes that achieve optimal limits—minimum storage regeneration (MSR) codes with optimal storage overhead and minimum bandwidth regeneration (MBR) codes with optimal repair bandwidth overhead [4]. Although reducing bandwidth overhead of repairing failed codes, the proposed regenerating codes don’t take into account disk I/O overhead. The disk I/O overhead is directly proportional to the number of surviving nodes connected in repair process [5]. In order to ensure fewer nodes connected in repair process, Papailiopoulos and Dimakis proposed locally repairable codes (LRC) [6]. Combining RS codes with XOR operations, Papailiopoulos et al. proposed a class of simple locally repairable codes, termed as simple regenerating codes (SRC) [7].

However, regenerating codes and LRC involve a large number of operations over finite field GF(q) in repair process, that is to say, the computational complexity in repair process is too high. To further reduce computational complexity of repairing failed nodes, El Rouayheb and Ramchandran proposed Fractional Repetition (FR) codes, as a new class of MBR codes [8]. The FR codes can tolerate low-complexity uncoded repair for multiple failed nodes, with minimum repair bandwidth overhead and disk I/O overhead, thus performance of repairing node failures is improved significantly.

The traditional FR codes are mainly applied to static distributed storage systems, which storage overhead and repetition degree of coded packets remain unchanged. However, due to some faults occur at any time in distributed storage systems, including partial data loss of storage nodes and storage node failures, the actual distributed storage system is dynamic [9]. Aiming at the problem that traditional FR codes cannot adapt to dynamic storage system changes, Zhu proposed adaptive FR codes and construction method based on symmetric designs [9]. When partial data of adaptive FR codes is lost, the remaining data still satisfies the properties of FR codes without reconfiguring the system, which saves a lot of bandwidth overhead [10]. Subsequently, Olmez and Ramamoorthy constructed FR codes based on combinatorial design, and proposed resolvable FR codes constructed based on resolvable designs [11-13]. When nodes fail in dynamic distributed storage systems, resolvable FR codes can changes repetition degree by simply adjusting surviving nodes. Considering the changes of storage overhead and repetition degree of coded packets in dynamic distributed storage systems, based on circulant permutation matrices (CPMs) and affine permutation matrices (APMs), Su proposed adaptive-and-resolvable FR codes [14]. For large-scale distributed storage systems, the construction scheme involves a large number of operations. The FR codes only adapt to dynamic homogeneous storage systems, which storage overhead of nodes and repetition degree of coded packets are consistent. Thus the above FR codes cannot be applied to heterogeneous distributed storage systems.

For large-scale dynamic distributed storage systems, we propose a construction scheme of adaptive-and-resolvable FR codes based on hypergraph coloring in this paper. These FR codes constructed can be flexibly applied to dynamic distributed storage systems, where the node storage overhead, number of storage nodes, and repetition degree of coded packets can change randomly and dynamically. By proposing the existence condition of linear uniform regular hypergraph, this paper further confirms the existence condition of adaptive-and-resolvable FR codes. Moreover, the adaptive-and-resolvable FR codes can be easily constructed for any specified parameters in distributed storage system. Specifically, based on the heuristic algorithm of hypergraph coloring proposed in this paper, the linear uniform regular hypergraph is constructed. The edges and vertices in hypergraph correspond to nodes and coded packets of FR code respectively, further, the FR code is constructed. Due to different popularity of original data packets and coded data packets, based on hypergraph coloring, we generalize the construction of adaptive-and-resolvable FR codes to heterogeneous distributed storage systems. Theoretical analysis shows that, according to hypergraph coloring, the FR codes can achieve rapid repair for multiple failed nodes. Compared with RS codes, SRC and LRC, the FR codes have best performance in repair locality, repair bandwidth overhead, computational complexity and time overhead during repairing failed nodes.

In this work, we mainly introduce adaptive-and-resolvable FR codes based on hypergraph coloring. In Section II, we introduce the preliminaries, mainly including hypergraph and adaptive-and-resolvable FR codes. The construction scheme based on hypergraph coloring is proposed in section III. In section IV, we study rapid repair method based on hypergraph coloring for single failed node and multiple failed nodes. In Section V, we generalize the construction of FR codes based on hypergraph coloring to heterogeneous distributed storage systems. Theoretical analysis and simulation results are shown in section VI. In Section VII, we draw a conclusion.

2. Preliminaries

2.1 Hypergraph

Definition 1 (hypergraph) [15]: As hypergraph H = (V, E), V is a finite set of vertices in hypergraph, and E is a set of non-empty subsets in V called edges in hypergraph. An edge can connect at least one vertex.

The hypergraph with no loops or multiple identical edges is termed as simple hypergraph [16]. The linear hypergraph is a simple hypergraph with at most one common vertex in any two edges. If each edge contains r vertices, the hypergraph is termed as r -uniform hypergraph. Similarly, if each vertex is contained by t edges, the hypergraph is termed as t -regular hypergraph. If linear hypergraph is both r-uniform hypergraph and t-regular hypergraph, the hypergraph is termed as linear r -uniform t-regular hypergraph, i.e. (r, t)-hypergraph [16].

Definition 2 (incidence matrix): A hypergraph H with vertices set V= {v₁, v₂, ..., v_n} and edges set E = {e₁, e₂, ..., e_m} has a n x m incidence matrix A = (a_ij)_nxm, where a_ij = 1 if v_i ∈ e_j, otherwise a_ij = 0.

For hypergraph H, if edges e_i and e_j(i ≠ j) contain the same vertex, the edges e_i and e_j are adjacent edges. Similarly, if vertices v_i and v_j (i ≠ j) are contained by the same edge, the vertices v_i and v_j are adjacent vertices.

Definition 3 (locally correlated edges): For hypergraph H, assumed that edges e_i and e_j(i ≠ j) only contain vertices v_i and v_j respectively, if vertices v_i and v_j are adjacent vertices, edges e_i and e_j are termed as locally correlated edges.

2.2 Adaptive-and-Resolvable FR Codes

In (n, k, d) distributed storage systems, assume that the number of storage nodes is n and the node storage overhead is α, the original files can be reconstructed by connecting any k storage nodes. And d surviving nodes can be connected to repair a single failed node. The above parameters satisfy k ≤ d ≤ n−1 and d = α.

Definition 4 (FR codes) [8]: In (n, k, d) distributed storage systems, FR code C = (Ω, N) is a set of n subsets, i.e. N = {N₁, ..., N_n}, the symbols in each subset belong to symbol set Ω = {1, ..., θ}, and repetition degree of C is ρ [17]. The (n, d, θ, ρ) FR code has two characteristics.

(a) The size of subset N_i is d.

(b) Each symbol in Ω belongs to ρ subsets in set N, and each pair of subsets contain one common symbol at most.

According to above definition, subset N_i represents the storage node of FR code C, the symbol in set Ω represents the coded packet by MDS encoding, and the elements in subset N_i represents the coded packets stored in storage node N_i. For FR code C, the amount of coded packets is θ, the repetition degree is ρ, all coded packets are stored in n storage nodes, and the storage overhead of nodes is d. Therefore, the parameters of FR code satisfy θρ = nd.

Definition 5 (Adaptive FR Codes) [9]: In FR code C = (Ω, N), if there is S ⊂ Ω and the set consisting of non-empty subsets N₀∖S, N₁∖S, ..., N_n-1∖S is an FR code, then C is an adaptive FR code.

In practical distributed storage systems, partial data loss of storage nodes is common phenomenon, especially the systems cannot be recovered for permanent failure. For adaptive FR code C, due to the remaining coded packets of storage nodes still satisfy the properties of FR codes, a new FR code C' can be obtained by simply adjusting FR code C. And FR code C' can be applied to new distributed storage system without reconfiguring the system.

Definition 6 (resolvable FR code): In FR code C = (Ω, N), as a subset P ⊂ N, if N_i ∈ P and N_j ∈ P(i ≠ j), there exists N_i I N_j = 𝜙 and U_{j:Vj∈P}N_j = Ω, then P is termed as a parallel class. If N consists of several parallel classes, then FR code C is termed as a resolvable FR code [13].

According to Definition 6, any two nodes in a parallel class do not contain the same coded packets, and all nodes in a parallel class contain all coded packets. There exist at least two parallel classes in resolvable FR code. The resolvable FR codes can dynamically change the repetition degree by adjusting parallel classes to adapt to dynamic distributed storage systems. Combining Definition 5 with Definition 6, the FR codes with both adaptive and resolvable properties are termed as adaptive-and-resolvable FR codes [14].

3. Adaptive-and-Resolvable FR Codes Based on Hypergraph Coloring

Consider that the (d, ρ)-hypergraph H = (V, E) with vertices set V = {v₁, v₂, ..., v_θ} and edges set E = {e₁, e₂, ..., e_n}, any two edges contain one same vertex at most.

The existence condition of (d, ρ)-hypergraph H is as follows:

(1) n ≡ 0 mod ρ ;

(2) n/ρ = θ/d or nd =θρ ;

(3) d² ≤ θ ;

(4) ρ² ≤ n.

The heuristic algorithm for hypergraph coloring is as follows:

Step 1: The vertices set V = {v₁, v₂, ..., v_θ} is divided into d vertices subsets by sequence, each subset contains θ/d vertices, which are V₁ = {v₁, ..., v_θ/d}, …, V_d = {v_(θ-θ/d+1), ..., v_θ} respectively. The edges set E = {e₁, e₂, ..., e_n} is divided into ρ edges subsets by sequence, which are E₁ = {e_1,1, ..., e_1,θ/d}, …, E_ρ = {e_ρ,1, ..., e_ρ,θ/d} respectively.

Step 2: For t =1, the indices of edges 1 ≤ j ≤ θ/d and the indices of vertices 1 ≤ i ≤ θ, we assign vertices to the edges e_1,1, ..., e_1,θ/d. When i = j mod(θ/d) , the vertex {v_i|i = j mod(θ/d), 1 ≤ i≤ θ} is assigned to the edge e_i,j.

Step 3: For 2 ≤ t ≤ ρ and the indices of edges 1≤ j ≤ θ/d, according to the vertices assigned to edges e_1,1, ..., e_1,θ/d, ..., e_t-1,1, ..., e_t-1,θ/d, we assign vertices to edges e_t,1, ..., e_t,θ/d. This situation satisfies the following:

(1) The edges e_t,1, ..., e_t,θ/d contain all vertices and any two edges are not adjacent or locally correlated;

(2) There is only one vertex of each vertices subset V₁, ..., V_d in edge e_t,j (1 ≤ j ≤ θ/d);

(3) Any two vertices in vertices set V can is contained by one edge of e_1,1, ..., e_1,θ/d, …, e_t-1,1, ..., e_t-1,θ/d, e_t,1, ..., e_t,θ/d at most;

Step 4: Hypergraph coloring. For 1 ≤ t ≤ ρ, the edges subset {e_t,1, ..., e_t,θ/d} are assigned the same color. There are ρ different colors in the hypergraph, and each vertex in set V = {v₁, v₂, ..., v_θ} is contained by ρ edges of different colors.

The edges and vertices in hypergraph correspond to the storage nodes and coded packets of above FR codes respectively. The vertices assigned to edges in hypergraph correspond to the coded packets stored in nodes of FR codes. The edges with the same color in hypergraph correspond to a parallel class in FR codes. The (d, ρ)-hypergraph is constructed based on the heuristic algorithm of hypergraph coloring. Further, (n, d, θ, ρ) adaptive-and-resolvable FR codes can be constructed.

Example 1: For (4, 3)-hypergraph H = (V, E), vertices set V = {v₁, v₂, ..., v₁₆} and edges set E = {e₁, e₂, ..., e₁₂}. First, the vertices set V is divided into 4 vertices subsets by sequence, i.e. V₁ = {v₁, v₂, v₃, v₄}, V₂ = {v₅, v₆, v₇, v₈}, V₃ = {v₉, v₁₀, v₁₁, v₁₂}, and V₄ = {v₁₃, v₁₄, v₁₅, v₁₆} ; the edges set E is divided into 3 edges subsets, i.e. E₁ = {e₁, e₂, e₃, e₄}, E₂ = {e₅, e₆, e₇, e₈}, and E₃ = {e₉, e₁₀, e₁₁, e₁₂}.

As shown in Fig. 1, according to the heuristic algorithm of hypergraph coloring, let vertices are assigned to all edge subsets. The vertices contained by edges {e₁, e₂, e₃, e₄} are {v₁, v₅, v₉, v₁₃}, {v₂, v₆, v₁₀, v₁₄}, {v₃, v₇, v₁₁, v₁₅}, {v₄, v₈, v₁₂, v₁₆} respectively. The vertices contained by edges {e₅, e₆, e₇, e₈} are {v₁, v₆, v₁₁, v₁₆}, {v₂, v₅, v₁₂, v₁₅}, {v₃, v₈, v₉, v₁₄}, {v₄, v₇, v₁₀, v₁₃} respectively. The vertices contained by edge {e₉, e₁₀, e₁₁, e₁₂} are {v₁, v₇, v₁₂, v₁₄}, {v₂, v₈, v₁₁, v₁₃}, {v₃, v₅, v₁₀, v₁₆}, {v₄, v₆, v₉, v₁₅} respectively. Finally, the same edges subset is assigned the same color, so the chromatic number of all edges is 3.

Fig. 1. Based on heuristic algorithm of hypergraph coloring, the (d, ρ)-hypergraph is constructed.

According to the (4, 3)-hypergraph constructed in Example 1, the incidence matrix A is acquired. The vertices of hypergraph correspond to row vectors of incidence matrix, and edges of hypergraph correspond to column vectors of incidence matrix.

\(\begin{aligned}A=\left[\begin{array}{llllllllllll}1 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 1 \\ 1 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 1 & 0 \\ 0 & 1 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 1 \\ 0 & 0 & 1 & 0 & 0 & 0 & 0 & 1 & 1 & 0 & 0 & 0 \\ 0 & 0 & 0 & 1 & 0 & 0 & 1 & 0 & 0 & 1 & 0 & 0 \\ 1 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 1 \\ 0 & 1 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 1 & 0 \\ 0 & 0 & 1 & 0 & 1 & 0 & 0 & 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 1 & 0 & 1 & 0 & 0 & 1 & 0 & 0 & 0 \\ 1 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 1 & 0 & 0 \\ 0 & 1 & 0 & 0 & 0 & 0 & 1 & 0 & 1 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 1 \\ 0 & 0 & 0 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 1 & 0\end{array}\right]\end{aligned}\)

The above incidence matrix can also represent FR code. The row vectors of incidence matrix correspond to coded packets of FR code, and column vectors correspond to storage nodes of FR code. Similarly, the weight of row vectors represents the repetition degree ρ of coded packets, and the weight of column vectors represents storage overhead d of nodes. Thus, the (12, 4,16, 3) adaptive-and-resolvable FR code is constructed based on incidence matrix. As shown in Fig. 2, there are 3 parallel classes {N₁, N₂, N₃, N₄}, {N₅, N₆, N₇, N₈} and {N₉, N₁₀, N₁₁, N₁₂} in FR code, corresponding to 3 edges subsets E₁ = {e₁, e₂, e₃, e₄}, E₂ = {e₅, e₆, e₇, e₈} and E₃ = {e₉, e₁₀, e₁₁, e₁₂} in hypergraph.

Fig. 2. The (12, 4, 16, 3) adaptive-and-resolvable FR code is constructed.

4. Repair of Failed Nodes Based on Hypergraph Coloring

According to hypergraph coloring situation, the adaptive-and-resolvable FR codes can repair the failed nodes.

4.1 Repair of Single Failed Node

For a single failed node, the newcomer node collects d coded packets from the surviving nodes corresponding to one complete edge subset (i.e. the surviving nodes in intact parallel class), which achieves the exact uncoded repair.

As shown in Fig. 3, when node N₁ corresponding to edge e₁ fails, the newcomer node can connect surviving nodes {N₅, N₆, N₇, N₈} corresponding to complete edge subset {e₅, e₆, e₇, e₈} to achieve the exact uncoded repair. Similarly, the newcomer node can also connect surviving nodes {N₉, N₁₀, N₁₁, N₁₂} corresponding to complete edge subset {e₉, e₁₀, e₁₁, e₁₂} to achieve the exact uncoded repair.

Fig. 3. Repair single failed node.

4.2 Repair of Multiple Failed Nodes

The ρ parallel classes in adaptive-and-resolvable FR code correspond to ρ chromatic edges subsets in hypergraph. The multiple failed nodes correspond to multiple edges in hypergraph. There are the following two cases of multiple failed nodes.

For the edges corresponding to multiple failed nodes with at most ρ−1 colors (i.e. at least one complete edge subset exists in hypergraph), the newcomer nodes connect to surviving nodes corresponding to one complete edge subset to achieve the exact uncoded repair.

As shown in Fig. 4, when nodes N₁ and N₅ fail, the newcomer nodes can connect surviving nodes {N₉, N₁₀, N₁₁, N₁₂} corresponding to complete edge subset {e₉, e₁₀, e₁₁, e₁₂} to achieve the exact uncoded repair.

Fig. 4. For the edges corresponding to multiple failed nodes with at most ρ−1 colors, repair multiple failed nodes.

For the edges corresponding to multiple failed nodes with ρ colors and the number of failed nodes no more than n−k, if there are ρ failed nodes stored the same coded packet, the newcomer nodes can connect any k surviving nodes to reconstruct original file. If there are not ρ failed nodes stored the same coded packet, firstly find the vertices and edges corresponding to failed coded packets and failed nodes, then find the adjacent edges of the failed edges, finally the newcomer nodes can connect surviving nodes corresponding to adjacent edges to achieve exact uncoded repair.

As shown in Fig. 5, when nodes N₁, N₅ and N₁₀ fail, the newcomer nodes can connect surviving nodes {N₂, N₃, N₇, N₈, N₉, N₁₁} to achieve exact uncoded repair.

Fig. 5. For the edges corresponding to multiple failed nodes with ρ colors, the number of failed nodes no more than n − k, and no more than ρ failed nodes stored the same coded packet, repair multiple failed nodes.

5. Construction of FR Codes in Heterogeneous Storage Systems

We generalize the construction of FR codes from homogeneous storage systems to heterogeneous storage systems, where storage overhead d and repetition degree ρ can be different. Actually, the popularity of original data packets and coded data packets is different, it is desirable that the more popular packets will have higher repetition degrees in distributed storage systems. Thus, we propose a new idea based on hypergraph coloring, that the repetition degree of coded data packets is reduced when the repetition degree of original data packets remains unchanged. Based on (d, ρ)-hypergraph, by deleting the edges in hypergraph corresponding to coded packets of adaptive-and-resolvable FR code, the FR code can be simply extended to heterogeneous distributed storage system.

Consider that the (n, d, θ, ρ) FR codes with a set of storage nodes N = {N₁, ..., N_n} and a set of coded packets Ω = {1, ..., θ} with repetition degree ρ. Assume that connecting any k storage nodes to download m coded packets can reconstruct original file. Then the set of original data packets is Ω₁ = {1, ..., m}, the set of coded data packets is Ω₂ = {m + 1, ..., θ}. According to the new idea proposed, the repetition degree ρ₁ of original data packets remains unchanged, that is ρ₁ = ρ, and the repetition degree ρ₂ of coded data packets is reduced, that is ρ₂ < ρ. Based on the heuristic algorithm of hypergraph coloring, when ρ₂ < t ≤ ρ, we delete the edges between the vertices {v_m+1, ..., v_θ} and the other vertices, then FR codes in heterogeneous storage system can be constructed. For (12, 4,16, 3) adaptive-and-resolvable FR code with repetition degree ρ = 3 and k = 4, the set of original data packets is Ω₁ = {1, ..., 11}, and the set of coded data packets is Ω₂ = {12, ...,16}. We specify that the repetition degree of original data packets is ρ₁ = 3 and that of coded data packets is ρ₂ = 2. Based on the heuristic algorithm of hypergraph coloring, the hypergraph is constructed as shown in Fig. 6(a), and the FR code in heterogeneous storage system is shown in Fig. 6(b).

Fig. 6. The construction of FR codes is generalized from homogeneous storage systems to heterogeneous storage systems.

6. Performance Analysis

For adaptive-and-resolvable FR codes based on hypergraph, performance is mainly analyzed from three aspects: repair locality, bandwidth overhead, computational complexity and repairing time. Moreover, the performance is compared with the common RS codes, SRC and LRC. In distributed storage systems, it is assumed that the number of storage nodes is n, the size of original file is B, and the reconstruction degree to RS codes, SRC and LRC is k. In the section, we only consider the cases of single node and two nodes failures.

6.1 Repair Locality

When a single node fails, as (n, k) RS codes, k surviving nodes need to be connected, so the repair locality is k. As (n, k, f) SRC, the original file consists of f subfiles, each subfile is encoded by RS encoding, then 2f surviving nodes need to be connected to repair the failed data, so the repair locality is 2f. As LRC, the original file is divided into r × k data packets, then r x n coded packets are obtained by (n, k) RS encoding, and the n check packets are obtained by XOR operations of coded packets with the same subscript. The coded packets and check packets are stored in r +1 nodes of each local repair group according to the subscript cycle. Then r surviving nodes in the local repair group need to be connected, so the repair locality is r. As adaptive-and-resolvable FR codes, d surviving nodes in a parallel class need to be connected, so the repair locality is d.

When two nodes fail simultaneously, as (n, k) RS codes and (n, k, f) SRC, the original file needs to be reconstructed by connecting k surviving nodes, so the repair localities are k. As LRC, when stored in different local repair groups, two failed nodes can be repaired, which is equivalent to a single node fails in each local repair group, so the repair locality is 2r. As adaptive-and-resolvable FR codes with ρ > 2, there are two cases on repairing for two failed nodes. (1) For two failed nodes in same parallel class, obviously, two failed nodes don’t store same coded packet, the min{2d, n/ρ} surviving nodes are connected of same parallel class. (2) For two failed nodes in different parallel classes, two failed nodes store at most one same coded packet, the min{2d - 1, n/ρ} surviving nodes are connected of same parallel class. When two nodes fail simultaneously, the maximum repair locality of adaptive-and-resolvable FR codes is min{2d, n/ρ}.

Assuming that reconstruction degree of original file to (n, k) RS codes, (n, k, f) SRC and LRC is k = n - 3. As (n, k, f) SRC, the original file consists of f = 3 subfiles, and each subfile is divided into k data packets. Similarly, as LRC, the original file is divided into 3k data packets, that is to say, when a single node fails, the repair locality is r=3. For the convenience of performance analysis, adaptive-and-resolvable FR codes is externally encoded by (n, k) MDS, that is, the number of coded packets is θ=n, and the repetition degree is ρ=3, so the node storage capacity is d=3. As shown in Fig. 7, whether a single node or two nodes fail, the adaptive-and-resolvable FR codes and LRC, have lower repair locality.

Fig. 7. The performance analysis is about repair locality.

6.2 Repair Bandwidth Overhead

When a single node fails, as (n, k) RS codes, repair locality is k, and storage overhead of nodes is B/k, so repair bandwidth overhead is B. As (n, k, f) SRC, f coded packets need to be download to repair one coded packet, the storage overhead of nodes is (f + 1)B/fk, so repair bandwidth overhead is (f + 1)B/k. Similarly, as LRC, repair bandwidth overhead is (r + 1)B/k. As adaptive-and-resolvable FR codes, the number of original data packets is j, the number of coded data packets is θ. When a single node fails, the d surviving nodes need to be connected, and one coded packet from each surviving node need to be downloaded, so repair bandwidth overhead is Bd/j.

When two nodes fail simultaneously, as (n, k) RS codes and (n, k, f) SRC, the original file needs to be reconstructed by connecting k surviving nodes, so their repair bandwidth overheads are B. As LRC, it is equivalent to repair one failed node in each local repair group, so repair bandwidth overhead is 2(r + 1)B/k . As adaptive-and-resolvable FR codes with ρ > 2, there are two cases on repairing for two failed nodes. (1) For two failed nodes in same parallel class, repair bandwidth overhead is 2Bd/j. (2) For two failed nodes in different parallel classes, repair bandwidth overhead is B(2d-1)/j.

Assume that the size of original file is B =1000Mb, encoding parameters are described in above subsection. As shown in Fig. 8, whether a single node or two nodes fail, the adaptive-and-resolvable FR codes have lower repair bandwidth overhead.

Fig. 8. The performance analysis is about repair bandwidth overhead.

6.3 Computational Complexity and Repair Time

As (n, k) RS codes, during the process of repairing failed nodes, any k surviving nodes need to be connected to reconstruct original file, then coded packets of failed nodes can be recovered by encoding to original file. Each coded data packet is obtained by operations between k original data packets on finite field GF(q). The decoding process of reconstructing original file is equivalent to encoding process. Thus the repair process of RS codes involves k² + k-times multiplication operations and k²−1-times addition operations over the finite field GF(q). As (n, k, f) SRC, a coded packet can be repaired by f−1-times XOR operation, and each node stores f + 1 coded packets. Thus the repair process of SRC involves (f − 1)(f + 1)-times XOR operations. Similar to SRC, the repair process of LRC involves (r − 1)(r + 1)-times XOR operations. As adaptive-and-resolvable FR codes, the corresponding surviving nodes are connected based on repair table, the newcomer nodes can download the failed coded packets directly. Thus the repair process of the FR codes does not involve any operations over finite field GF(q).

As shown in Table 1, adaptive-and-resolvable FR codes have the lowest computational complexity during repairing failed nodes.

Table 1. The computational complexity

When the failed nodes are repaired in distributed storage systems, RS codes involve a large number of operations over the finite field GF(q), with high computational complexity and long repair time. SRC and LRC involve some simple XOR operations, which also increases computational complexity and long repair time to some extent. The uncoded repair process of adaptive-and-resolvable FR codes only involves reading for coded packets, so computational complexity is the lowest, and it greatly reduces the repair time of failed nodes.

7. Conclusion

The new construction scheme of adaptive-and-resolvable FR codes based on hypergraph coloring has been proposed in this paper. These FR codes constructed can be flexibly applied to dynamic distributed storage systems, where the node storage overhead, number of storage nodes, and repetition degree of coded packets change randomly and dynamically. In this paper, the existence condition of (d, ρ)-hypergraph and the heuristic algorithm of hypergraph coloring are proposed. According to specified parameters in distributed storage system, (d, ρ)-hypergraph and corresponding (n, d, θ, ρ) adaptive-and-resolvable FR can be constructed based on the heuristic algorithm of hypergraph coloring. Based on hypergraph coloring, this paper analyzes the rapid repair methods for failed nodes. Further, the FR codes can be generalized to heterogeneous distributed storage systems based on hypergraph coloring. Compared with RS codes, SRC and LRC, adaptive-and-resolvable FR codes have better repair locality and repair bandwidth overhead. Moreover, adaptive-and-resolvable FR codes have great advantages on computational complexity and time overhead during repairing failed nodes.