# On Secure Distributed Data Storage Under Repair Dynamics

## Abstract

We address the problem of securing distributed storage systems against passive eavesdroppers that can observe a limited number of storage nodes. An important aspect of these systems is node failures over time, which demand a repair mechanism aimed at maintaining a targeted high level of system reliability. If an eavesdropper observes a node that is added to the system to replace a failed node, it will have access to all the data downloaded during repair, which can potentially compromise the entire information in the system. We are interested in determining the secrecy capacity of distributed storage systems under repair dynamics, i.e., the maximum amount of data that can be securely stored and made available to a legitimate user without revealing any information to any eavesdropper. We derive a general upper bound on the secrecy capacity and show that this bound is tight for the bandwidth-limited regime which is of importance in scenarios such as peer-to-peer distributed storage systems. We also provide a simple explicit code construction that achieves the capacity for this regime.

^{1}

## I Introduction

Data storage devices have evolved significantly since the days of punched cards. Nevertheless, storage devices, such as hard disks or flash drives, are still bound to fail after long periods of usage, risking the loss of valuable data. To solve this problem and to increase the reliability of the stored data, multiple storage nodes can be networked together to redundantly store the data, thus forming a distributed data storage system. Applications of such systems are innumerable and include large data centers and peer-to-peer storage systems, such as OceanStore [1], that use a large number of nodes spread widely across the Internet to store files.

Codes for protecting data from erasures have been well studied in classical channel coding theory, and can be used here to increase the reliability of distributed storage systems. Fig. 1 illustrates an example where a maximal distance separable (MDS) code is used to store a file of 4 symbols, , distributively on nodes, , each of capacity symbols. The MDS code implemented here ensures that any user, also called data collector, connecting to any storage nodes can obtain the whole file . However, what distinguishes the scenario here from the erasure channel counterpart is that when a storage node fails, it needs to be repaired or replaced by a new node in order to maintain a desired level of system reliability. A straightforward repair mechanism would be to add a new replacement node of capacity 2, and make it act as a data collector by connecting to surviving nodes. The new node can then download the whole file (4 symbols) to construct the lost part of the data and store it. Another repair scheme that consumes less bandwidth is depicted in Fig. 1 where node fails and is replaced by node . When node connect to 3 nodes instead of 2, it is possible to decrease the total repair bandwidth from 4 to 3 symbols. Note that does not need to store the exact data that was on ; the only required property is that the data stored on all the active nodes and form an MDS code.

The above important observations were the basis of the original work of [2] where the authors showed that there exists a fundamental tradeoff between the storage capacity of each node and the repair bandwidth. They also introduced and constructed “regenerating codes” as a new class of codes that generalize classical erasure codes and permit the operation of a distributed storage system at any point on the tradeoff curve.

When a distributed data storage system is formed using nodes widely spread across the Internet, e.g., Internet based peer-to-peer systems, individual nodes may not be secure and can become susceptible to eavesdropping. This paper focuses on such scenarios where an eavesdropper can gain access to a certain number of the storage nodes. The compromised distributed storage system is always assumed to be dynamic with nodes continually failing and being repaired. Thus, the compromised nodes can belong to the original set of storage nodes that the system starts with, or even include some of the replacement nodes added to the system to repair it from failures. Under this setting, we are interested in determining how much data can still be stored in the system without revealing any information to any of the eavesdroppers.

To answer this question, we follow the approach of [2] and model the distributed storage system as a multicast network that uses network coding. Under this model, the eavesdropper is an intruder that can access a fixed number of the network nodes of her choice. This eavesdropper model is natural for distributed storage systems and comes in contrast with the wiretapper model studied in the network coding literature [3, 4, 5] where the intruder can observe network edges, instead of nodes. We derive a general upper bound on the secrecy capacity as a function of the node storage capacity and the repair bandwidth. Motivated by system considerations, we define an important operating regime, that we call the bandwidth-limited regime, where the repair bandwidth is constrained not to exceed a given upper bound, while no limitation is imposed on the storage capacity of the nodes. For this important operating regime, we show that our upper bound is tight and present capacity-achieving codes.

This paper is organized as follows. In Section II we describe the system and security model. We define the problem and give a summary of our results in Section III. In Section IV we illustrate two special cases of distributed storage systems that are instructive in understanding the general problem. In Section V, we derive an upper bound on the secrecy capacity, and in Section VI, we present a scheme that achieves this upper bound for the case of bandwidth-limited regime. We conclude in Section VII.

## Ii Model

### Ii-a Distributed storage system

A distributed storage system (DSS) is a collection of storage nodes that includes a source node , that has an incompressible data file of symbols, or units, each belonging to a finite field . The source node is connected to storage nodes each with a storage capacity of symbols, which may be utilized to save coded parts of the file . The storage nodes are individually unreliable and may fail over time. To guarantee a certain desired level of reliability, we assume that the DSS is required to always have active, i.e., non-failed, storage nodes that are in service. Therefore, when a storage node fails, it is immediately replaced by a new node with same storage capacity . The DSS should be designed in a way to allow any legitimate user, that we also call data collector, that connects to any out of the active storage nodes available at any given time, to be able to reconstruct the original file . We term this condition as the “reconstruction property” of distributed storage systems.

We assume that nodes fail one at a time, and we denote by the new replacement node added to the system to repair the -th failure. The new replacement node connects then to some nodes, chosen randomly, out of the remaining active nodes and downloads units from them in total, which corresponds to the repair bandwidth of the system. The repair degree is a system parameter satisfying . In this work, we focus on the case of symmetrical repair where the new node downloads equal amount of data, say units, from each of the nodes it connects to, i.e., . The process of replenishing redundancy to maintain the reliability of a DSS is referred to as the “regeneration” or “repair” process. Note that a new replacement node may download more data than what it actually stores. Moreover, the stored data can possibly be different than the one that was stored on the failed node, as long as the “reconstruction property” of the DSS is retained. A distributed storage system is thus characterized as . For instance, the DSS depicted in Fig. 1 corresponds to which is operating at .

### Ii-B Flow Graph Representation

We adopt the flow graph model introduced in [2] which we describe here for completeness. In this model, the distributed storage system is represented by an information flow graph . The graph is a directed acyclic graph with capacity constrained edges that consists of three kinds of nodes: a single source node , input storage nodes and output storage nodes and data collectors DC for . The source node has an information of which a specific realization is the file . Each storage node in the DSS is represented by two nodes and joined by a directed edge of capacity (see Fig. 2), to account for the node storage constraint.

The repair process is initiated every time a failure occurs. As a result, the DSS, and consequently the flow graph, are dynamic and evolve with time. At any given time, each node in the graph is either active or inactive depending on whether it has failed or not. The graph starts with only the source node being active and connected to the storage input nodes by outgoing edges of infinite capacity. From this point onwards, the source node becomes and remains inactive and the input and output storage nodes become active. When a node fails in a DSS, the corresponding nodes and become inactive in . If a replacement node joins the DSS in the process of repairing a failure and connects to active nodes , the corresponding nodes and , with the edge , are added to the flow graph , and node is connected to the nodes by incoming edges of capacity each. A data collector is represented by a node connected to active storage output nodes through infinite capacity links enabling it to reconstruct the file . The graph constitutes a multicast network with the data collectors as destinations. An underlying assumption here is that the flow graph corresponding to a distributed storage system depends on the sequence of failed nodes. As an example, we depict in Fig. 2 the flow graph corresponding to the DSS of Fig. 1, when node fails.

### Ii-C Eavesdropper Model

We assume the presence of an intruder “Eve” in the DSS, who can observe up to , nodes of her choice among all the storage nodes, possibly at different time instances as the system evolves. In the flow graph model, Eve is an eavesdropper who can access a fixed number of nodes chosen from the storage input nodes . Notice that while a data collector observes output storage nodes, i.e., the data stored on the nodes it connects to, Eve, has access to input storage nodes, and thus can observe, in addition to the stored data, all incoming messages to these nodes. We also assume that Eve has complete knowledge of the storage and repair schemes implemented in the DSS. Thus, she can choose some of the nodes to be among the initial storage nodes, or, if she deems it more profitable, she can choose to wait for failures and eavesdrop on a replacement node by observing its downloaded data. Eve is assumed to be passive, and only observes the data without modifying it.

## Iii Problem Statement and Results

### Iii-a Secrecy Capacity

Let be a random vector uniformly distributed over , representing the incompressible data file at the source node with . Let and be the sets of input and output storage nodes in respectively. For a storage node , let and be the random variables representing its downloaded messages and stored content respectively. Thus, , represents the data that can be downloaded by a data collector when contacting node , while , with , represents the total data revealed to Eve when she accesses node . The stored data is a function of the downloaded data .

Let be the collection of all subsets of of cardinality consisting of nodes that are simultaneously active at some instant in time. For any subset of , define . Similarly, for any subset of , define . The reconstruction property, then, can be written as

(1) |

and the perfect secrecy condition implies

(2) |

Given a DSS with compromised nodes, its secrecy capacity, denoted by ), is then defined to be the maximum amount of data that can be stored in this system such that the reconstruction property and the perfect secrecy condition are simultaneously satisfied for all possible data collectors and eavesdroppers i.e.,

(3) |

where , and

### Iii-B Results

First, we give the following general upper bound on the secrecy capacity of a DSS:

###### Theorem 1

[Upper Bound] For a distributed data storage system , with a repair degree , and compromised nodes, the secrecy capacity is upper bounded as

(4) |

where .

Next, we consider an important operational regime, namely the bandwidth-limited regime, where the repair bandwidth is constrained to a maximum amount , i.e., , while no constraint is imposed on the storage capacity at each node. The secrecy capacity in this regime is defined as,

(5) |

For a fixed , when the parameter is a system design choice, the upper bound of Theorem 1 on the secrecy capacity can be further optimized, and attains a maximum for . In section VI, we demonstrate that this upper bound can be achieved for in the bandwidth-limited regime. Thus, establishing the following theorem:

###### Theorem 2

[Bandwidth-Limited Regime] For a distributed data storage system , compromised nodes, the secrecy capacity for a bandwidth-limited regime, for , is

(6) |

and is achieved with a storage capacity of .

## Iv Special Cases

### Iv-a Static Systems

A static version of the problem studied here corresponds to a DSS with ideal storage nodes that do not fail. Hence there is no need for any repair in the system. The flow graph of this system is then the combination network studied in network coding theory (see for e.g. [6, Chap. 4] ). Therefore, the static storage problem can be regarded as a special case of wiretap networks [3, 4], or equivalently, as the erasure-erasure wiretap-II channel studied in [7]. The secrecy capacity for such systems is , and can be achieved using either nested MDS codes [7], or the coset codes of [8, 4].

Even though the above proposed solution is optimal for the static case, it can have a very poor secrecy performance when applied directly to dynamic storage systems with failures. For instance, a straightforward way to repair a failed node would be to download the whole file on the new replacement node, and then generate the specific lost data. In this case, if Eve accesses the new replacement node while it is downloading the whole file, it will be able to reconstruct the entire original data. Hence, the secrecy rate for this scheme would be zero. However, Theorem 2 suggests that for some systems we can achieve a positive secrecy capacity. This example highlights the fact that dynamical repair of the DSS renders it intrinsically different from the static counterpart, and one should be careful in designing the repair scheme in order to safeguard the whole stored data.

### Iv-B Systems Using Random Network Coding

Using the flow graph model, the authors of [2] showed that random linear network codes over a large finite field can achieve any point , on the optimal storage-repair bandwidth tradeoff curve with a high probability. Consider an example of random linear network code used in a compromised DSS , which stores symbols and operates at and . In this case, each of the initial nodes stores independently generated random linear combinations of these symbols. Assume now that node fails and is replaced by a new node that connects to , and , and downloads from each one of them random linear combination of their stored data. Assume that after some time, node fails and is replaced by node in a similar fashion. Now, if , and Eve accesses nodes and while they were being repaired, it will observe 6 linear combinations of the original data symbols, which, with high probability are linearly independent. Therefore, she will be able to reconstruct the whole file.

The above analysis shows that, when random network coding is used, it is not possible to achieve a positive secrecy rate for this system, even with pre-processing at the source, using for example Maximum Rank Distance (MRD) codes [5]. But according to Theorem 2, which we prove in section VI, the secrecy capacity of the the above DSS is equal to one unit when . This is also in contrast with the case of multicast networks with compromised edges instead of nodes [3], wherein, random network coding can perform as good as any deterministic secure code [5].

## V Upper bound on secrecy capacity

In this section we derive the upper bound of Theorem 1. Consider a DSS with . Assume that the nodes have failed consecutively, and were replaced during the repair process by the nodes respectively as shown in Fig. 3. Now suppose that Eve accesses nodes in while they were being repaired, and consider a data collector connected to the nodes in . The reconstruction property implies by Eq. (1), and the perfect secrecy condition implies by Eq. (2). We can therefore write

Inequality follows from the fact that the stored data is a function of the downloaded data , (2) from, , (3) follows from the fact that each node can store at most units, and for each replacement node we have , also from the topology of the network (see Fig. 3). Note that each node is connected to each of the nodes by an edge of capacity . The upper bound of Theorem 1 follows then directly from the definition of Eq. (3).

## Vi Secrecy Capacity in the Bandwidth-Limited Regime

### Vi-a Example

Consider again the DSS with , and of Section IV-B, for which the secrecy rate using random linear network coding was shown to be . The upper bound on the secrecy capacity of this system given by Theorem 1 is . We provide a scheme that achieves this upper bound. The proposed code is depicted in Fig. 4 and consists of the concatenation of an MDS coset code [8] with a special repetition code that was introduced in [9] by Rashmi et al. for constructing exact regeneration codes. Let denote the information symbol to be securely stored on the system. is encoded using the outer MDS code into a codeword , where are independent random keys uniformly distributed over and . The encoded symbols are then stored on the nodes as shown in Fig. 4, following the special repetition code of [9]. It is easy to verify that any data collector connecting to nodes, observes all the symbols , and can therefore decode . However, an eavesdropper accessing any two nodes will only observe symbols out of , and cannot gain any information about . Next, we generalize this construction to obtain a capacity-achieving code for the bandwidth-limited regime.

### Vi-B Code Construction

Our approach builds on the results of [9] where the authors constructed a family of exact regenerating codes for . The “exact” property of these codes allows any repair node to reconstruct and store an identical copy of the data lost upon a failure. For simplicity, we will explain the construction for , i.e., . For any larger values of , and in turn of , the file can be split into chunks, each of which can be separately encoded using the construction corresponding to . Choose . From [2] we know that is the capacity of the above DSS in the absence of any adversary (). Let be the number of information symbols that we would like to store securely on the DSS, and . Let denote the information file and denote independent random keys each uniformly distributed over . Then, the proposed code consists of an outer nested MDS coset code [7] which takes and as an input, and outputs , such that , where is a generator matrix of a MDS code, and in itself is a generator matrix for a MDS code. The information vector effectively selects the coset of the MDS code generated by .

This outer MDS code is then followed by the special repetition code introduced in [9] which stores the codeword on the DSS. The procedure of constructing this inner code can be described using an auxiliary complete graph over vertices that consists of edges. Suppose the edges are indexed by the coded symbols . The code then consists of storing on node the indices of the edges adjacent to vertex in the complete graph. Consequently, every coded symbol is stored on exactly two storage nodes, and any pair of two storage nodes have exactly one distinct coded symbol in common, e.g., code in Fig. 4 for .

This inner code transforms the dynamic storage system into an equivalent static
point-to-point channel. First notice that , hence
all the data downloaded during the repair process, i.e., , is stored on the new replacement node without any further
compression. Thus, accessing a node during repair process, i.e.,
observing its downloaded data, is equivalent to accessing it after
the repair process, i.e., observing its stored data. Second, the
exact regeneration codes restore a failed node with the exact lost
data. So, even though there are failures and repairs, the data storage
system looks exactly the same at any point of time. Any data
collector downloads symbols out of by
connecting to nodes. Moreover, any eavesdropper can observe
symbols. Thus, the system
becomes similar to the erasure-erasure wiretap channel-II of
parameters ^{2}

## Vii Conclusion

In this paper we considered dynamic distributed data storage systems that are subject to eavesdropping. Our main objective was to determine the secrecy capacity of such systems, i.e., the maximum amount of data that these systems can store and deliver to data collectors, without revealing any information to the eavesdropper. Modeling such systems as multicast networks with compromised nodes, we gave an upper bound on the secrecy capacity, and showed that it can be achieved in the important bandwidth-limited regime where the nodes have sufficient storage capacity. Finding the general expression of the secrecy capacity of distributed storage systems, and more generally of multicast networks with a fixed number of compromised nodes, remains an open problem that we hope to address in future work.

### Footnotes

- This research was funded in part by an AFOSR grant (FA9550-09-1-0120), a DTRA grant (HDTRA1-09-1-0032), and an NSF grant (CCF-0830788).
- In the erasure-erasure wiretap channel-II of parameters , the transmitter sends symbols. A legitimate receiver and an eavesdropper receive and symbols respectively through independent erasure channels [7].

### References

- S. Rhea, C. Wells, P. Eaton, D. Geels, B. Zhao, H. Weatherspoon, and J. Kubiatowicz, “Maintenance-free global data storage,” IEEE Internet Computing, pp. 40–49, 2001.
- A. Dimakis, P. Godfrey, Y. Wu, M. Wainright, and K. Ramchandran, “Network coding for distributed storage systems,” to appear in IEEE Trans. Inform. Theory.
- N. Cai and R. W. Yeung, “Secure network coding,” in Proc. IEEE Int. Symp. Inf. Theory (ISIT), 2002.
- S. El Rouayheb and E. Soljanin, “On wiretap networks II,” in Proc. IEEE Int. Symp. Inf. Theory (ISIT), (Nice, France), 2007.
- D. Silva and F. Kschischang, “Security for wiretap networks via rank-metric codes,” in Proc. IEEE Int. Symp. Inf. Theory (ISIT), 2008.
- R. Yeung, S.-Y. Li, and N. Cai, Network Coding Theory (Foundations and Trends in Communications and Information Theory). Now Publishers Inc, 2006.
- S. Arunkumar and S. W. Mclaughlin, “MDS codes on erasure-erasure wire-tap channel,” in arXiv:0902.3286v1, 2009.
- L. H. Ozarow and A. D. Wyner, “Wire-tap channel-II,” in AT&T Bell lab tech. journal vol. 63, no. 10, 1984.
- K. Rashmi, N. B. Shah, P. V. Kumar, and K. Ramchandran, “Exact regenerating codes for distributed storage,” in Allerton Conference on Control, Computing, and Communication, Urbana-Champaign, IL, 2009.