The demand for storage is growing rapidly every year, and businesses and consumers are storing unprecedented amounts of data. IDC predicts worldwide data will grow to 175 zettabytes by 2025, with as much data stored in the cloud as in data centers¹.
The Storj network introduces an economy around storage by creating a decentralized storage platform which enables anyone to become a storage node operator. Storage nodes aren’t managed by Storj Labs and storage node operators are peers in the community managing hardware and networks that Storj Labs doesn’t control. Thus, the Storj network uniquely enables the combination of storage from any cloud and any data center into one global storage system. This combination of storage results in a decentralized network² and increased challenges in some key areas of scale, security, and performance.
Minimizing coordination between computers is vital to any large scale system design. This is even truer given the diversity of the hardware and network environments in which the storage nodes in the Storj network live. The number of storage nodes involved in any Storj file upload or download make preventing slow or misbehaving nodes from impacting response times essential. The Storj network applies redundancy and coordination avoidance to counter long tail latency or high throughput demands to deliver a healthy network for its users.
Redundancy
The Storj network uses erasure codes³ to store segments of your file across the network to provide a high level of durability. These erasure codes are spread many times across many machines so the network can tolerate failures without losing data. This redundancy also provides a performance side effect; if during any upload or download a set of storage nodes are responding slowly, the network will ignore these slow storage nodes and complete the upload or download using the responses from the faster storage nodes. As long as enough of the redundant nodes have responded with all the required segments, the network acknowledges the user request and discards the slow responses in favor of the responses from the faster nodes—this prevents long tail latency from impacting user response times.
Coordination Avoidance
Blockchain consensus offers very strong guarantees, but this comes at a heavy cost in coordination overhead. Bailis et al.’s⁴ describes how coordination is not always necessary for correctness and minimizing coordination is key to maximizing scalability, availability, and high performance in database systems. One fundamental design decision of the Storj network was not to utilize blockchain consensus for file transfers to increase those properties of the Storj network. Storj takes a pragmatic approach to avoiding blockchain consensus while still maintaining correctness for file transfers. But, at the same time by default, Storj uses blockchain consensus with the Ethereum-based STORJ token for payment processing to storage node operators.
We recently announced Tardigrade[⁵](https://storj.io/blog/2019/04/introducing-tardigrade
