Error Resilience in Distributed Storagevia Rank-Metric Codes

Error Resilience in Distributed Storage
via Rank-Metric Codes

Natalia Silberstein, Ankit Singh Rawat and Sriram Vishwanath LINC, Dept. of ECE, UT Austin
Austin, TX 78712, USA.
Email: {natalys, ankitsr, sriram}@austin.utexas.edu
Abstract

This paper presents a novel coding scheme for distributed storage systems containing nodes with adversarial errors. The key challenge in such systems is the propagation of erroneous data from a single corrupted node to the rest of the system during a node repair process. This paper presents a concatenated coding scheme which is based on two types of codes: maximum rank distance (MRD) code as an outer code and optimal repair maximal distance separable (MDS) array code as an inner code. Given this, two different types of adversarial errors are considered: the first type considers an adversary that can replace the content of an affected node only once; while the second attack-type considers an adversary that can pollute data an unbounded number of times. This paper proves that the proposed coding scheme attains a suitable upper bound on resilience capacity for the first type of error. Further, the paper presents mechanisms that combine this code with subspace signatures to achieve error resilience for the second type of errors. Finally, the paper concludes by presenting a construction based on MRD codes for optimal locally repairable scalar codes that can tolerate adversarial errors.

I Introduction

Distributed storage systems (DSS) are increasingly important resources today for users and businesses than ever before. Centralized storage is proving to be rapidly infeasible given the growing sizes of databases being stored; as well as the need to rapidly and reliably access them. Thus, generally, data is distributed and often replicated across multiple locations to enable ease-of-access and reslience to failures. However, replication can be inefficient in terms of storage space, and therefore, coding is useful in instilling resilience to node failures while reducing storage requirements over replication. Given the prevalence of single node failures in DSS (a user exiting a P2P system, power outage in a single data center in the cloud), coding enables a single node to be repaired as soon as the node failure occurs in order to sustain the desired level of redundancy in the system.

In order to repair the failed node, data is downloaded from surviving nodes and a function of this data is stored as the ‘restored’ node. The amount of data downloaded in this repair process is called the repair bandwidth of the reconstruction process. Since large repair bandwidth translate to consumption of a vast amount of system resources (in terms of bandwidth and energy), it is desirable to have a coding schemes that have as small a repair bandwidth as possible In [1], Dimakis et al. establish an information theoretic lower bound on repair bandwidth for codes with maximum distance separable (MDS) property (i.e., any set of nodes can be used to reconstruct the data stored on a DSS). The work also presents a trade-off between repair bandwidth and the amount of data stored on each node.

A desirable notion of repair is exact repair, where the regenerated data is an exact replica of what was stored on the failed node. The work in [2, 3, 4] present low rate codes, which achieve the lower bound derived in [1] when data is downloaded from all surviving nodes during node repair. In [5], Rashmi et al. design exact-repairable codes, which allow node repair to be performed by contacting surviving nodes. At the minimum storage regeneration (MSR) point of operation (see [1] for details), these codes also correspond to a low rate regime as their rate is upper bounded by . Recently, various researchers have presented high(er) rate codes for the MSR point which enable bandwidth-efficient exact repair. Along a similar view, [6] presents codes for DSS with two parity nodes. In [7] and [8], permutation-matrix based codes are presented which are designed to achieve the bound on repair bandwidth for repair of systematic nodes for all pairs. [9] further generalizes the idea of [8] to get MDS array codes for DSS that allow optimal exact regeneration for parity nodes as well.

While a bulk of existing literature in DSS addresses the storage versus repair bandwidth trade-off, another important issue that has recently received attention is the design of storage schemes that ensure security and reliability of the stored content against adversarial errors [10, 11, 12, 13]; and this issue of inducing reliability against adversarial errors is the main focus of our paper. The dynamic nature of DSS resulting from repeated node failures, triggering node repairs, makes the problem of dealing with erroneous nodes non-trivial. This is because a single corrupted node may subsequently pollute other nodes in the DSS system during the node repair process. In [12], Pawar et al. address the reliability issue in detail and derive upper bounds on the amount of data that can be stored on the system and reliably made available to a data collector when bandwidth optimal node repair is performed. [12] also presents coding strategies that achieve the upper bound for a particular range of system parameters, namely in the bandwidth-limited regime. A related but different problem of securing stored data against passive eavesdroppers is addressed in [12, 14].

In this paper, we study the notion of an omniscient adversary111In [13], Rashmi et al. show that product matrix codes [5] can be used towards dealing with adversarial errors. This solution is again restricted to the regime , at the MSR point of operation., which can observe all nodes and has full knowledge of the coding scheme employed by the system [12]. As in [12], we assume an upper bound on the number of nodes that can be controlled by such an omniscient adversary. We classify adversarial attacks into two classes:

  1. Static errors: an omniscient adversary replaces the content of an affected node with nonsensical information only once. The affected node uses this same polluted information during all subsequent repair and data collection processes. Static errors represent a common type of data corruption due to wear out of storage devices, such as latent disk errors or other physical defects of the storage media, where the data stored on a node is permanently distorted.

  2. Dynamic errors: an omniscient adversary may replace the content of an affected node, each time the node is asked for the data during data collection or repair process. This kind of errors captures any malicious behaviour, hence is more difficult to manage in comparison to static errors.

We present a novel concatenated coding scheme for DSS which provides resilience against these two classes of attacks. In our scheme, the content to be stored is first encoded using a maximum rank distance (MRD) code. The output of this outer code is further encoded using a maximum distance separable (MDS) array code, which can be repaired in a bandwidth efficient manner. Using an MRD code, which is an optimal rank-metric code, allows us to quantify the errors introduced in the system using their rank as opposed to their Hamming weights. As stated before, the dynamic nature of the DSS causes a large number of nodes to get polluted even by a single erroneous node, as false information spreads from node repairs. Thus, a single polluted node infects many others, resulting in an error vector with a large Hamming weight. Using rank-metric codes can help alleviate this problem as the error that a data collector has to handle has a known rank, and can therefore be corrected by an MRD code with a sufficient rank distance. Using an bandwidth efficient MDS array code as inner code facilitates bandwidth efficient node repair in the event of a single node failure and allows the data collector to recover the original data from any subset of storage nodes. In this paper, we use exact-regenerating bandwidth efficient codes operating at the minimum-storage regenerating (MSR) point [1]. However, our construction works for any regenerating code.

The proposed coding scheme is directly applicable to the static error model and is optimal in terms of amount of data that can be reliably stored on a DSS under static error model. The model with dynamic errors is more complicated, as it allows a single malicious node to change its pollution pattern, and introduce an arbitrarily large error both in Hamming weight and in rank. For dynamic error model, we propose two solutions based on the concatenated coding scheme employed for static error model. One solution is to exploit the inherent redundancy in the encoded data due to outer code, i.e., an MRD code, and perform error free node repair even in the presence of adversarial nodes. This solution, namely naïve method, is optimal for a specific choice of parameters. Alternative solution combines our concatenated coding scheme with subspace signature based cryptographic schemes to control the amount (rank) of pollution (error) introduced by an adversarial node. We employ the signature scheme by Zhao et al. [15], which essentially reduces the dynamic error model to somewhat similar to static error model, and helps us bound the rank of error introduced by an adversarial node throughout its presence in the system. Note that hash function based solutions has previously been presented in the context of DSS to deal with errors [10, 12]. While promising, these hash functions based approaches provide only probabilistic guarantees for pollution containment.

Recently a new line of work has appeared in the context of DSS, which focuses on minimizing the number of surviving nodes that contribute data during node repair (locality). In [16], Gopalan et al. establish an upper bound on the minimum distance of scalar linear codes that have locality . They further show the optimality of pyramid codes [18] with respect to the upper bound, with locality of the systematic nodes. [19] gives a similar bound for codes that have a more general definition of locality. In [20], Papailiopoulos et al. generalize the bound in [16] to vector codes (possibly nonlinear). In this paper, we also show that MRD codes can be utilized to construct optimal (in terms of minimum distance) locally repairable scalar codes, which can tolerate adversarial errors.

The rest of the paper is organized as follows: In Section II, we first give a brief description of rank-metric codes along with Gabidulin MRD codes and the error model in rank-metric for these codes. Subsequently, we describe MDS array codes and present two examples of bandwidth efficient MDS array codes that are later used as inner codes in our construction. In Section III, we describe the construction of our storage scheme and prove its error resilience under the static error model. Further, we present a few examples to illustrate our scheme and show that our codes attain the upper bound on resilience capability. In Section IV, we address the dynamic error model. In Section V, we describe MRD codes based locally repairable storage schemes. Finally, we conclude with Section VI.

Ii Preliminaries

Ii-a System Model

Let be the size of a file to be stored in a distributed storage system with nodes. All data symbols belong to a finite field of size . Each node contains symbols. A data collector reconstructs the original file by downloading the data stored in any set of out of nodes. This property is called an MDS property. When a node fails, its content can be reconstructed by downloading symbols from any surviving nodes and stored in a new node added to DSS instead of the failed node. The amount of data needed for a node repair, denoted by , is called the repair bandwidth. In [1], Dimakis et al. show a trade-off between repair bandwidth and the amount of data stored on each node. Two families of codes which attain two extremal points on the optimal tradeoff curve are called minimum-storage regenerating (MSR) codes and minimum-bandwidth regenerating (MBR) codes, respectively. In the sequel we will focuss on the MSR codes. It was proved [1] that for MSR codes and satisfy

(1)

Ii-B Rank-Metric Codes

Rank-metric codes were introduced by Delsarte [21] and rediscovered in [22, 23]. These codes have applications in different fields, such as space-time coding [24], random network coding [25, 26], and public key cryptosystems [27]. Our goal in this paper is to show that rank-metric codes are useful for error correction in distributed storage as well.

Let be the finite field of size . For two matrices and over the rank distance is defined by

An rank-metric code is a linear code, whose codewords are matrices over ; they form a linear subspace with dimension of , and for each two distinct codewords and , . For an rank-metric code we have  [21, 22, 23]. This bound, called Singleton bound for rank metric, is achievable for all feasible parameters. The codes that achieve this bound are called maximum rank distance (MRD) codes.

Ii-B1 Gabidulin MRD Codes

An important family of MRD linear codes is presented by Gabidulin [22]. These codes can be seen as the analogs of Reed-Solomon codes for rank metric. Let . A codeword in an rank-metric code can be represented by a vector , where , since can be viewed as an -dimensional vector space over . Let , , be linearly independent over . The generator matrix of an Gabidulin MRD code is given by

where , , and .

In the similar way as Reed-Solomon codes, Gabidulin codes also have an equivalent interpretation as evaluation of polynomials, however, for Gabidulin codes the special family of polynomials, called linearized polynomials, is evaluated on a set of linearly independent points over the base field . A linearized polynomial over of -degree has the form , where , and . It is easy to see that evaluation of a linearized polynomial is an -linear transformation from to itself, i.e., for any and , we have  [28]. Therefore, a codeword in Gabidulin code can be defined as , where is the linearized polynomial of -degree with coefficients given by the information message, and are linearly independent over  [22].

Ii-B2 Rank Errors and Rank Erasures Correction

Let be a Gabidulin MRD code with minimum distance . Let be the transmitted codeword and let be the received word. The code can correct any vector error of the form as long as . The first part is called a rank error of rank , where are linearly independent over the base field , unknown to the decoder, and are linearly independent vectors of length , unknown to the decoder. The second part is called a rank erasure, where are linearly independent over the base field , unknown to the decoder, and are linearly independent vectors of length , and known to the decoder. Note, that in the matrix form we can write

(2)

[22, 29] present decoding algorithms for rank-metric codes.

(a)
(b)
Fig. 1: Illustration of the second node repair process in Zigzag code: (a) for error free system, (b) for system with erroneous information at the first storage node.

Ii-C MDS Array Codes for Distributed Storage

A linear array code of dimensions over is defined as a linear subspace of . Its minimum distance is defined as the minimum Hamming distance over , when we consider the codewords of as vectors of length over . An array code is called an maximum distance separable (MDS) code if , where  [30, 31]. Note, that an MRD code is also an MDS array code.

Let be an information vector, is a block of size , for all . These blocks are encoded into encoded blocks , , stored in nodes of size , in the following way:

where and the generator matrix is an block matrix with blocks of size given by:

An array code has an MDS property if any blocks sub matrix of of size is of the full rank. In other words, if an MDS code is used to store data in a system, and any set of storage nodes fails, the original data can be recovered from the surviving nodes.

We say that an MDS code satisfies optimal repair property, if a single failed node can be repaired by downloading elements from every node of any -subset of surviving nodes [1].

Ii-C1 Examples of Optimal Repair MDS Array Codes

In the following, we present two examples of the optimal repair MDS array codes for DSS, which we will use further for illustration of our coding scheme. Due to space constraints, we only describe the MDS codes used in the examples. For general constructions, interested readers may refer to the respective papers that present these codes.

Example 1

(5,3) Zigzag code [8]. This class of MDS array codes [8] is based on generalized permutation matrices. For the Zigzag code presented in Fig. 1, the first three nodes are systematic nodes which store the data , with . The block generator matrix for this code is given by

where and 0 denote the identity matrix and all-zero matrix, respectively. Fig. (a)a describes node repair process for Zigzag code. When the second node fails, the newcomer node downloads the symbols from the shaded locations at the surviving nodes.

Example 2

Hadamard Design codes [6]. This class of MDS array codes employs interference alignment strategies in order to perform node repair. In the example presented in Fig. 2, the first three nodes are systematic nodes which store the data, , . The block generator matrix for a Hadamard design based code is given by

where , , , , and and satisfy . The process of the second node repair is illustrated in Fig. (a)a. During this process, the newcomer downloads and from node and , respectively (see in Fig. (a)a). The newcomer uses and as repair matrices corresponding to node and , respectively, where and are some basis for the column-space of and , respectively. The information downloaded from node and is used to cancel the interference terms (contribution of and in ). After interference mitigation, a linear system of equations is solved to get .

Iii Code Construction for Static Error Model

In this section we present our coding scheme and prove its error tolerance under the static error model. We illustrate the idea by using examples from Sec. II-C1 and prove that this construction is optimal for the static error model.

Iii-a The Construction

Let denote a file of size , . is partitioned into parts of size each. We form an matrix over , where an th part of forms an th column of , for all . Let be an Gabidulin MRD code, with . Let be the codeword in which corresponds to the information matrix . Let be positive integers such that and let . Let be an optimal repair MDS array code of dimensions . We partition the vector into parts of size and form nodes of size each, according to the encoding algorithm of the code . Note, that we use an MDS array code over , i.e., its generator matrix is over , and during the process of node repair, a set of surviving nodes transmits linear combinations of the stored elements with the coefficients from .

The following theorem shows that if the system tolerates up to erroneous nodes.

Theorem 1

Let be the number of erroneous nodes in the system based on concatenated MRD and optimal repair MDS array codes. If , then the original data can be recovered from any nodes.

{proof}

Let be the codeword in which corresponds to an information matrix , and let , , be the partition of into parts of size each. Let , be the encoded blocks stored in nodes.

Let be the set of indices of the erroneous nodes. Hence the th node, , contains , where denotes an adversarial error introduced by the th node. When the failed nodes are being repaired, the errors from adversarial nodes propagate to the repaired nodes. In particular, th node, , contains , where represents the propagation of error and depends on the specific choice of an MDS array code. Suppose a data collector contacts a subset with nodes and downloads from any th node, . If these nodes are all systematic nodes, then we obtain , where is the blocks matrix of size over given by

and . Otherwise, we obtain , where the block matrix represents the coefficients of obtained by the decoding of the code . Since the rank of over is at most , and , the MRD code can correct this error.

(a)
(b)
Fig. 2: Illustration of node repair in Hadamard design based codes: (a) in error free system, (b) in the presence of error at the first storage node.

Iii-B Examples

In this subsection we illustrate the idea of the construction for the case where an adversary pollutes the information stored on a single storage node. We demonstrate that the rank of the error introduced by an adversary does not increase due to node repair dynamics under the static error model. Hence, a data collector can recover the correct original information using a decoder for an MRD code. It is important to note that in our construction, any optimal repair MDS array code from DSS literature can be used as the inner code. In this subsection, we illustrate the idea of our construction with the help of two examples drawn from two different classes of optimal repair MDS array codes for DSS, presented in Section II-C1.

Example 3

Let be the (5,3) Zigzag code from Example 1. Its first three systematic nodes store a codeword , , from Gabidulin MRD code, which is obtained by encoding the original data. The content stored in th systematic node, , is . Note that in this example. Let us assume that an adversary attacks the first storage node and introduces erroneous information. The erroneous information at the first node can be modeled as . Now assume that the second node fails. The system is oblivious to the presence of pollution at the first node, and employs an exact regeneration strategy to reconstruct the second node. The reconstructed node downloads the symbols from the shaded locations at the surviving nodes, as described in Fig. (b)b, and solves a linear system of equations to obtain , where denotes the inverse element of in , . Now assume that a data collector accesses the first three nodes in an attempt to recover the original data. The data collector now has access to , where

(3)

Note that contains an error of rank at most four. Therefore, the original MRD codeword and subsequently the original information can be recovered, using an MRD code with rank distance at least nine.

Example 4

Let be a (5,3) Hadamard design based code, described in Example 2. Its first three nodes store , , where is a codeword of a Gabidulin MRD code, which is obtained by encoding the original data. Suppose an adversary modifies the information stored at the first node to . When the second node fails, a newcomer, unaware of the presence of error at the first node, employs the interference alignment based strategy described in Example 2 and depicted in Fig. 2. After interference mitigation, a linear system of equations is solved to obtain . Assuming that a data collector contacts the first three nodes, it receives which contains an error of rank at most . This allows the recovery of uncorrupted information using an MRD code of sufficient minimum rank distance.

Iii-C Code Parameters

The upper bound on the amount of data that can be stored reliably in the general system with corrupted nodes, called resilience capacity and denoted by , was presented by Pawar et al. [12]. This bound is given by

(4)

The authors in  [12] provided the explicit construction of the codes that attain this bound, for bandwidth-limited regime. However, this construction has practical limitations for large values of since the decoding algorithm presented in [12] is exponential in .

Next, we show that our constructed codes attain bound (4) and thus, are optimal. In addition, the decoding of codewords in the construction presented in our paper is efficient since it is based on two efficient decoding algorithms: one, for an MDS array code, and two, for a Gabidulin code.

Let the parameters be as described in the construction of Section III. Then and . Let be an integer such that . Then , and hence .

Now we compare this result with the bound (4). Let be an MDS array code with optimal repair property. Then . Therefore, we can rewrite bound (4) as follows:

Thus, the coding scheme proposed in this paper achieves the bound in (4).

Remark 1

Although the static error model is less general than the model considered in [12], the upper bound (4) applies to the static model as well. Pawar et al. [12] obtain this upper bound by evaluating a cut of the information flow graph corresponding to a particular node failure sequence, pattern of nodes under adversarial attack and data collector. This information flow graph is also valid in the context of the static error model. Consequently its cut, which provides an upper bound on the information flow and represents the amount of data that can be reliably stored on DSS, is correct under the static model. As shown in this paper, this bound is tight for static model at MSR point, which might not be the case for the general error model considered in [12].

Remark 2

Recently, Rashmi et al. [13] considered a scenario, referred as ‘erasure’, where some nodes which are supposed to provide data during node repair become unavailable. It is easy to see from (2) that our construction can also correct such erasures, as long as the minimum distance of the corresponding MRD code is large enough. The codes obtained by our construction also attain the bound on the capacity derived in [13]. However, while our construction works with any MSR code and in particular with an MSR code with high rate, it provides a solution for a restricted error model.

Iv Dynamic Error Model

In this section, we consider the problem of designing coding schemes for DSS that work under dynamic error model. In this attack model, an adversary can have access to at most storage nodes throughout the life span of DSS and can potentially send different data each time a node under its control is meant to send data during node repair as well as data reconstruction process.

Note that coding scheme proposed for the static error model hinges on the fact that each time an attacked node is requested for the data to be sent, it sends some linear combinations of the data that has been modified on it by adversary, which the adversary is allowed to do only once. Therefore, the rank of the error that a single node under static attack causes, throughout the operation of DSS, is bounded above by . This is not the case under the dynamic error model as a single attacked node can inject an error of large rank if it is utilized in multiple node repairs, which may render the data stored on DSS useless.

Towards this model, some results are presented in [12] and [13]. As we have discussed earlier, the coding scheme proposed in [12] does not have an efficient decoding during the data reconstruction process and it works specifically with bandwidth efficiently repairable codes at MBR point, however we focus on MSR point in this paper. The coding scheme of [13] deals with the dynamic error model at MSR point, but there scheme works only for low rate, i.e., . In coding scheme proposed in [13], an adversarial node does not inject errors into any other node during node repair, as this scheme allows newcomer to perform error-free exact repair.

Next, we present two solutions to deal with attack under the dynamic error model. The first solution aims to correct errors during the node repair process by using a simple approach, which is explained in the following subsection. The second approach to cope with dynamic errors is based on existing literature on subspace signatures.

Iv-a Naïve Scheme for Dynamic Error Model

In the dynamic error model, where there is no constraint on an adversarial nodes in terms of the data that it can provide during node repair as well as data reconstruction, one solution might be to design coding schemes that correct errors while node repair. One such scheme is proposed in [13] for low rate codes, where a newcomer node utilizes the redundancy in the downloaded data to perform correct exact repair even in the presence of errors in the downloaded data. Next, we analyze the maximum amount of information that can be stored on the DSS employing concatenated codes proposed in Section III under the dynamic error model, if an error free node repair needs to be performed.

When a storage node fails, a newcomer node downloads symbols from any surviving nodes . Since there can be at most adversarial nodes present in the system, the newcomer node receives at most erroneous symbols. Therefore, out of symbols of an MRD codeword, by (1) the newcomer has symbols (using the fact that the inner code is an MDS code and we perform bandwidth efficient repair). All the other symbols of an MRD codeword can be considered as the erased symbols. Let denote the number of information symbols (over ) that are stored on the DSS. Then the minimum distance of the corresponding MRD code satisfies . Therefore we can reconstruct the entire MRD codeword and thus the data stored on the failed node, if we have

This gives us

(5)

Note that the bound in (4) is still applicable. For , the right hand side expression in (5) is equal to that in (4). Therefore, this naïve repair scheme is optimal in terms of the capacity of DSS even in the dynamic error model. However, the difference between these bounds is monotonically increasing with and the solution proposed in this section is suboptimal for general values of system parameters and .

Iv-B Subspace Signatures Approach

As mentioned previously, in dynamic error model an attacked node can inject a high rank error. Thus, it is desirable to restrict the rank of the aggregate error that a particular attacked node can cause in the entire system under dynamic error model. In this subsection, we propose to combine the existing literature on detecting subspace pollution with MRD codes to counter a dynamic attack. Next, we illustrate this with the help of subspace signatures proposed in [15].

Let us consider an -nodes DSS, which employs an MRD and a bandwidth efficiently repairable code based storage scheme as explained in Section III. For a node , content stored on it, i.e. can be viewed as an matrix over . These vectors of length stored on node span a subspace (column space of when viewed as a matrix over ) in of rank at most . Since all elements of coding matrix and repair matrices are from , linear combinations sent by node for a node repair are nothing but, vectors that lie in the subspace spanned by . If we make sure that even under dynamic error model an attacked node sends vectors from the same -dimensional subspace of during node repair and data construction, a data collector encounters at most -rank error, which can be corrected with an MRD code of large enough rank distance as in the static model. Subspace signatures solve this problem of enforcing a node to send data (vectors) from the same -rank subspace of .

At the beginning, when data is encoded to be stored on a DSS, system administrator also generates subspace signatures for data stored on each node according to the procedure explained in [15]. We assume existence of a trusted verifier, who stores all subspace signatures, one signature for each storage node. Whenever a particular node sends data during node repair or data reconstruction, the truster verifier checks the data against the stored subspace signature corresponding to that particular storage node.

For the purpose of data reconstruction, whenever a node does not pass the signature test, that node is considered an rank erasure. If nodes fail the test during data reconstruction, the data collector deals with rank erasures and rank errors. This bound on the rank of the error that the data collector encounters is explained in the following paragraph. Given that the outer MRD code has minimum rank distance , the original data can be reconstructed without an error.

Next, we argue how subspace signatures help restrict rank of the error introduced in the system by adversarial nodes under dynamic error model. Assume that node fails. Let denote the set of surviving nodes that are contacted to repair node . In order to repair node , each node is supposed to send , where is an repair matrix of node associated with node . Since the data downloaded through all the surviving nodes is verified against subspace signatures, data from node passes the test if it is of the form , where is an matrix, which may be different form . Note that is in the column space of .

If any of the surviving nodes does not pass the test, the trusted verifier begins the naïve repair for the failed node and the nodes that fail the test. During this naïve repair, entire data is downloaded from a set of nodes out of nodes that provide data for node repair in the first place and pass the subspace test. Here is the number of nodes that fail the subspace test. Note that each node of these selected nodes provides additional symbols as it has already sent symbols (over ). The decoding algorithm for MRD codes is run on symbols downloaded from the selected set of nodes. There can be at most adversary nodes present in the selected set of nodes ( adversarial nodes that failed the subspace test are excluded from this process), which can contribute at most erroneous symbols. Since the rank distance of the MRD code is greater than , the decoding algorithm recovers the original file, which is used to get the data that is supposed to be stored on nodes being repaired.

In case when all the adversarial nodes pass the test, the data provided by each node is of the form

After performing exact repair process for node , node stores , where is an matrix over and . Here denotes the set of adversarial nodes. After the node repair, the trusted verifier generates a new subspace signature corresponding to the data stored on a node for future verification. At any point of time, the data stored on DSS can be represented as

(6)

where an th column of is equal to , . It is evident from (6) that the rank of the aggregate error in the system is at most and an MRD code with large enough minimum distance can ensure the reliable recovery of the original data.

The idea of using hash functions to provide tolerance against adversarial errors in DSS has previously been presented in [10]. However, the error tolerance scheme of [10] involves periodic verification of the data stored in the system which may not be practical.

V Error Resilience for Locally Repairable Codes

In this section we consider a special case where and each node can be repaired by accessing at most other nodes, where is called locality. Note that in this scenario there exist nodes for repair of a single node, and not every set of nodes can be used for a node repair. Such codes were considered in [16, 17, 19, 20]. Clearly, such codes do not have an MDS property, but they have a small repair bandwidth. It is proved [16] that the minimum distance of such a code satisfies

(7)

An explicit construction of codes which attain this bound with systematic locality is presented in [16, 18]. However an explicit construction for codes that attain this bound and have all symbols locality is only known for  [19]. In what follows we show a new construction of codes with all symbols locality which attain bound (7) and can also correct static errors. This construction is also based on MRD codes.

V-a Construction for Optimal Locally Repairable Codes

Let be an Gabidulin MRD code, where each codeword is considered as a vector of length over , and let be the given locality. Each symbol of an MRD codeword will be stored in a new node. We consider two cases: and , for . In the first case we partition the set of coordinates into disjoint groups , , of size , and in the second case we partition the set of coordinates into disjoint groups of size and one additional group of size . For each group we add a new parity node to be the sum of all the symbols in the same group. For the group the new parity symbol is equal to the sum of the elements of . We denote by the constructed code.

Example 5

Let , , , and be a codeword of an MRD code, . Then the corresponding codeword of is equal to , where , and .

Theorem 2

A code attains bound (7), i.e., it is an code with .

{proof}

As it was mentioned in Section II, a codeword of a Gabidulin code can be considered as an evaluation of a linearized polynomial on the set of linearly independent points over , . Note that for the reconstruction the original data we need at least values , such that are linearly independent over  [17]. By the linearity of we have that for any group , the new parity is equal to , for , and for we have , for . Hence, any symbols in are the evaluation of in linearly independent points. Now we distinguish between two cases.

Case 1, : In this case, . Let and be two integers such that , , and . Then , and according to bound (7), . We will prove that any erasures can be corrected by this code. In other words, after this number of erasures we still have an evaluation of in linearly independent points. Note that the worst case is when the erasures appear in the smallest possible number of groups and when the number of erasures inside a group is maximal. So we consider the case when all the symbols in groups are erased, and there is a group with erasures. Then the number of the remaining symbols which correspond to linearly independent points is .

Case 2, : for . In this case, . Let be an integer such that , . Since , then by bound (7), . As in the previous case, we consider the worst case for erasures, when all the symbols in groups and one additional symbol are erased. Then the number of the remaining symbols which correspond to linearly independent points is .

Example 6

In this example, we prove that the code from Example 5 has minimum distance , in other words, this code can tolerate any three erasures.

  • any erasures can be corrected by the MRD code, since its minimum rank distance is .

  • if two erasures are in the same group and additional one erasure in another group, then we still have the values of the corresponding linearized polynomial in the linearly independent points,

  • if all three erasures are in the same group, then we have values of in linearly independent points.

In all the cases, since , we can correct these erasures.

Note that if there is an adversarial node in such a system, then the erroneous data spreads into the whole group containing this node because of the node repair property, in other words, the Hamming weight of an error is equal to the size of a group. However, the rank of such an error is one, and it can be corrected by an MRD code with . We generalize this property of a code in the following theorem.

Theorem 3

If the related to MRD code has minimum rank distance then can tolerate at most static errors.

Example 7

Consider the code from Example 5, and let . Its rank distance is , and the Hamming distance is . Since the group size is , by using only Hamming metric it is impossible to correct this error, however, by applying a decoding of MRD code we can correct it.

Remark 3

The construction proposed in this section can be seen as a particular case of the construction of self-repairing homomorphic codes [17], since it can also be described in terms of linearized polynomials. In addition, this construction is closely related to construction of pyramid codes [18], and the construction given in [19], since an MRD code can be also seen as an MDS code over an extension field.

Vi Conclusion

A novel concatenated coding scheme for distributed storage system was presented. The scheme makes use of rank-metric codes, in particular, MRD codes as the first step of encoding the data. In the second step of encoding MDS optimal repair array codes or locally repairable codes are used. This construction provides resilience against static adversarial errors. Moreover, when using MDS optimal repair array codes this scheme is optimal in terms of the resilience capacity, and when using locally repairable codes this scheme is optimal in terms of the minimum distance. A modification of the scheme based on subspace signatures provides resilience against dynamic errors. The resilience of this scheme against a passive eavesdropper is a subject of our current research [32].

References

  • [1] A. G. Dimakis, P. Godfrey, M. Wainwright, and K. Ramachandran, “Network coding for distributed storage system”, in IEEE Transactions on Information Theory, vol. 56, no. 9, pp. 4539-4551, September 2010.
  • [2] Y. Wu and A. G. Dimakis, “Reducing repair traffic for erasure coding-based storage via interference alignment”, in Proc. of IEEE ISIT, 2009.
  • [3] N. B. Shah, K. V. Rashmi, P. V. Kumar and K. Ramchandran, “Explicit codes minimizing repair bandwidth for distributed storage”, in Proc. of IEEE ITW, Jan 2010.
  • [4] C. Suh and K. Ramchandran, “Exact-repair MDS codes for distributed storage using interference alignment”, in Proc. of IEEE ISIT, 2010.
  • [5] K. V. Rashmi, N. B. Shah and P. V. Kumar, “Optimal Exact-regenerating Codes for Distributed Storage at the MSR and MBR point via a Product-Matrix Construction”, IEEE Transactions on Information Theory, vol. 57, no. 57, pp. 5227-5239, Aug. 2011.
  • [6] D. Papailiopoulos and A. Dimakis, “Repair optimal erasure codes through hadamard designs,” in Proc. of Allerton, 2011.
  • [7] V. Cadambe, C. Huang and J. Li, “Permutation code: Optimal exact-repair of a single failed node in mds code based distributed storage systems,” in Proc. of IEEE ISIT, 2011.
  • [8] I. Tamo, Z. Wang, and J. Bruck, “Zigzag codes: MDS array codes with optimal rebuilding,” CoRR, vol. abs/1112.0371, 2011.
  • [9] Z. Wang, I. Tamo and J. Bruck, “On Codes for Optimal Rebuilding Access”, in Proc. Allerton Conf. on Control, Computing and Communication, Sep. 2011.
  • [10] T. K. Dikaliotis, A. G. Dimakis, T. Ho, “Security in distributed storage systems by communicating a logarithmic number of bits”, in Proc. of ISIT, 2010.
  • [11] F. Oggier and A. Datta, “Byzantine Fault Tolerance of Regenerating Codes”, in Proc. of IEEE Int. Conf. on Peer-to-Peer Computing, 2011.
  • [12] S.  Pawar, S. El Rouayheb and K. Ramchandran, “Securing Dynamic Distributed Storage Systems against Eavesdropping and Adversarial Attacks”, IEEE Trans. Inform. Theory , vol. 57, no. 9, Sep. 2011.
  • [13] K. V. Rashmi. N. B. Shah, K. Ramchandran, and P. V. Kumar, “Regenerating codes for errors and erasures in distributed storage”, CoRR, vol. abs/1202.1050, 2012.
  • [14] N. B. Shah, K. V. Rashmi and P. V. Kumar, “Information-theoretically secure Regenerating Codes for Distributed Storage”, in Proc. of IEEE Globecom, Dec. 2011.
  • [15] F. Zhao, T. Kalker, M. Medard, and K. J. Han, “Signatures for content distribution with network coding”, in Proc. of ISIT, June 2007.
  • [16] P. Gopalan, C. Huang, H. Simitchi, and S. Yekhanin, “On the locality of codeword symbols”, CoRR, vol. abs/1106.3625, 2011.
  • [17] F. Oggier and A. Datta, “Homomorphic self-repairung codes for agile maintenance of distributed storage systems”, CoRR, vol. abs/1107.3129, 2011.
  • [18] C. Huang, M. Chen, and J. Li, “Pyramid code: Flexible schemes to trade space for access efficiency in relible data storage systems”, in 6th IEEE nternational symposium on Network Computing and Applications (NCA 2007),pp.79–86, 2007.
  • [19] N. Prakash, G. M. Kamath, V. Lalitha, and P. V. Kumar, “Optimal linear codes with a local-error-correction property”, CoRR, vol. abs/1107.3129, 2011.
  • [20] D.S. Papailiopoulos and A.G. Dimakis, “Locally Repairable Codes”, to appear in ISIT 2012
  • [21] P. Delsarte, “Bilinear forms over a finite field, with applications to coding theory,” Journal of Comb. Theory, Series A, vol. 25, pp. 226-241, 1978.
  • [22] E. M. Gabidulin, “Theory of codes with maximum rank distance,” Problems of Information Transmission, vol. 21, pp. 1-12, July 1985.
  • [23] R. M. Roth, “Maximum-rank array codes and their application to crisscross error correction,” IEEE Trans. Inform. Theory, vol. 37, pp. 328-336, March 1991.
  • [24] P. Lusina, E. Gabidulin, and M. Bossert,“Maximum Rank Distance codes as space-time codes,” IEEE Trans. Inform. Theory, vol. 49, pp. 2757– 2760, Oct. 2003.
  • [25] D. Silva, F. R. Kschischang, and R. Koetter, “A rank-metric approach to error control in random network coding,” IEEE Trans. Inform. Theory, vol. 54, pp. 3951–3967, Sep. 2008.
  • [26] T. Etzion and N. Silberstein, “Error-correcting codes in projective space via rank-metric codes and Ferrers diagrams”, IEEE Trans. Inform. Theory, vol. 55, no.7, pp. 2909–2919, July 2009.
  • [27] E. M. Gabidulin, A. V. Paramonov, and O. V. Tretjakov, “Ideals over a non-commutative ring and their application in cryptology,” LNCS, vol. 573, pp. 482�-489, 1991.
  • [28] F. J. MacWilliams and N. J. A. Sloane, The theory of error-correcting codes, North-Holland, 1978.
  • [29] D. Silva and F. R. Kschischang, “Fast Encoding and Decoding of Gabidulin Codes.” in Proc. of IEEE ISIT, 2009.
  • [30] M. Blaum and R. M. Roth, “On lowest density MDS codes”, IEEE Trans. Inform. Theory, vol. 45, pp. 46–59, 1999.
  • [31] Y. Cassuto and J. Bruck,“Cyclic low-density MDS array codes”, in Proc. of IEEE ISIT, 2006.
  • [32] O. O. Koyluoglu, A. S. Rawat, N. Silberstein, and S. Vishwanath, ”On secrecy in distributed storage systems”, in preparation.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
48353
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description