Decentralization Storage: The Evolution from Concept Hype to Practical Implementation

Storage used to be one of the hot tracks in the blockchain industry. Filecoin, as the leader of the last bull market, once had a market cap exceeding $10 billion. Arweave, with permanent storage as its selling point, reached a maximum market cap of $3.5 billion. However, as the feasibility of cold data storage is questioned, the necessity of permanent storage is also challenged, making it a big question whether decentralized storage can truly succeed. The emergence of Walrus brings new vitality to the long-silent storage narrative, and now Aptos and Jump Crypto have launched Shelby, aiming to enhance the application of decentralized storage in the hot data field. So, can decentralized storage make a comeback and provide a wide range of application scenarios? Or is it just another hype? This article will analyze the evolution of the decentralized storage narrative from the development paths of four projects: Filecoin, Arweave, Walrus, and Shelby, and explore how far decentralized storage still needs to go to achieve widespread adoption.

Filecoin: Surface Storage, Actual Mining

Filecoin is one of the early rising token projects, and its development direction naturally revolves around Decentralization. This was a common characteristic of altcoins at the time - seeking the meaning of Decentralization in various traditional fields. Filecoin is no exception; it connects storage with Decentralization and points out the drawbacks of centralized data storage: the trust assumption on centralized storage service providers. Therefore, Filecoin attempts to transform centralized storage into Decentralized storage. However, certain aspects sacrificed for the sake of Decentralization later became pain points that projects like Arweave or Walrus attempted to address. To understand why Filecoin is essentially just a mining coin, one needs to recognize the objective limitations of its underlying technology, IPFS, which is not suitable for handling hot data.

IPFS: Decentralization architecture, limited by transmission bottlenecks

IPFS( InterPlanetary File System) was introduced around 2015, aiming to disrupt the traditional HTTP protocol through content addressing. The biggest drawback of IPFS is its extremely slow retrieval speed. In an era where traditional data service providers can achieve millisecond-level responses, retrieving a file with IPFS still takes tens of seconds, making it difficult to promote in practical applications and explaining why it is rarely adopted by traditional industries, except for a few blockchain projects.

The underlying P2P protocol of IPFS is primarily suited for "cold data", which refers to static content that does not change frequently, such as videos, images, and documents. However, when it comes to handling hot data, such as dynamic web pages, online games, or artificial intelligence applications, the P2P protocol does not have a significant advantage over traditional CDNs.

Although IPFS itself is not a blockchain, its design concept of directed acyclic graph (DAG) is highly compatible with many public chains and Web3 protocols, making it naturally suitable as a foundational building framework for blockchains. Therefore, even if it lacks practical value, it is sufficient as a foundational framework for carrying blockchain narratives. Early altcoin projects only needed a functioning framework to launch grand visions, but as Filecoin developed to a certain stage, the inherent flaws brought by IPFS began to hinder its progress.

Mining logic under the storage cloak

The original intention of IPFS was to allow users to become part of the storage network while storing data. However, without economic incentives, it is difficult for users to voluntarily use this system, let alone become active storage nodes. This means that most users will only store files on IPFS but will not contribute their storage space or store others' files. It is against this backdrop that Filecoin was born.

In the Filecoin token economic model, there are three main roles: users are responsible for paying fees to store data; storage miners receive token incentives for storing user data; retrieval miners provide data when users need it and receive incentives.

This model has potential malicious space. Storage miners may fill garbage data after providing storage space to obtain rewards. Since this garbage data will not be retrieved, even if lost, it will not trigger the penalty mechanism for storage miners. This allows storage miners to delete garbage data and repeat this process. Filecoin's Proof of Replication consensus can only ensure that user data has not been privately deleted, but it cannot prevent miners from filling garbage data.

The operation of Filecoin largely relies on miners' continuous investment in the token economy, rather than on the real demand from end users for distributed storage. Although the project is still undergoing iterations, at this stage, the ecological construction of Filecoin aligns more with the "mining logic" rather than the "application-driven" definition of storage projects.

Arweave: The Success and Failure of Long-termism

If Filecoin's design goal is to build an incentivized, verifiable Decentralization "data cloud" shell, then Arweave takes an extreme direction in storage: providing the capability for permanent storage of data. Arweave does not attempt to build a distributed computing platform; its entire system unfolds around a core assumption - important data should be stored once and forever retained in the network. This extreme long-termism makes Arweave vastly different from Filecoin in terms of mechanisms, incentive models, hardware requirements, and narrative perspectives.

Arweave uses Bitcoin as a learning object, attempting to continuously optimize its permanent storage network over long periods measured in years. Arweave does not care about marketing, nor does it care about competitors and market trends. It simply continues to iterate on its network architecture, moving forward even if no one pays attention, because this is the essence of the Arweave development team: long-termism. Thanks to long-termism, Arweave was highly sought after during the last bull market; also because of long-termism, even if it falls to the bottom, Arweave may still withstand several rounds of bull and bear markets. The only question is whether future decentralized storage will have a place for Arweave. The value of permanent storage can only be proven through time.

Since the Arweave mainnet version 1.5 to the recent version 2.9, despite losing market discussion, it has been committed to allowing a broader range of miners to participate in the network at minimal cost and incentivizing miners to maximize data storage, thereby continuously enhancing the robustness of the entire network. Arweave is well aware that it does not align with market preferences, thus adopting a conservative approach, not embracing the miner community, resulting in a complete stagnation of the ecosystem, upgrading the mainnet at minimal cost, and continuously lowering the hardware threshold without compromising network security.

A Review of the Upgrade Path from 1.5 to 2.9

The Arweave 1.5 version exposed a vulnerability where miners could rely on GPU stacking instead of real storage to optimize block production chances. To curb this trend, version 1.7 introduced the RandomX algorithm, which limits the use of specialized computing power and requires general CPUs to participate in mining, thereby weakening computing power centralization.

In version 2.0, Arweave adopts SPoA, transforming data proofs into a concise path of Merkle tree structure, and introduces format 2 transactions to reduce synchronization burdens. This architecture alleviates network bandwidth pressure and significantly enhances node collaboration capabilities. However, some miners can still evade the responsibility of holding real data through centralized high-speed storage pool strategies.

To correct this bias, version 2.4 introduced the SPoRA mechanism, which incorporates global indexing and slow hash random access, requiring miners to genuinely hold data blocks to participate in valid block production, thereby weakening the effect of hashpower stacking from a mechanism perspective. As a result, miners began to focus on storage access speed, driving the application of SSDs and high-speed read-write devices. Version 2.6 introduced hash chain control for block production rhythm, balancing the marginal benefits of high-performance devices and providing a fair participation space for small and medium miners.

Subsequent versions further enhance network collaboration capabilities and storage diversity: 2.7 adds collaborative mining and pool mechanisms to improve the competitiveness of small miners; 2.8 launches a composite packaging mechanism that allows large-capacity low-speed devices to participate flexibly; 2.9 introduces a new packaging process in replica_2_9 format, significantly increasing efficiency and reducing computational dependencies, completing the closed loop of the data-driven mining model.

Overall, Arweave's upgrade path clearly presents its storage-oriented long-term strategy: while continuously resisting the trend of computational power centralization, it lowers the participation threshold to ensure the long-term viability of the protocol.

Walrus: Is embracing hot data hype or is there a deeper meaning?

The design concept of Walrus is completely different from that of Filecoin and Arweave. The starting point of Filecoin is to create a decentralized and verifiable storage system, at the cost of cold data storage; the starting point of Arweave is to create an on-chain library of Alexandria that can permanently store data, at the cost of too few scenarios; the starting point of Walrus is to optimize the storage overhead of hot data storage protocols.

Magic Modified Error Correction Code: Cost Innovation or Old Wine in New Bottles?

In terms of storage cost design, Walrus believes that the storage overhead of Filecoin and Arweave is unreasonable, as both latter systems adopt a fully replicated architecture. Their main advantage lies in each node holding a complete copy, providing strong fault tolerance and independence among nodes. This type of architecture ensures that even if some nodes go offline, the network still maintains data availability. However, this also means that the system requires multiple copies for redundancy to maintain robustness, which in turn increases storage costs. Particularly in Arweave's design, the consensus mechanism itself encourages node redundancy for enhanced data security. In contrast, Filecoin is more flexible in cost control, but at the expense of potentially higher data loss risks with some low-cost storage options. Walrus attempts to find a balance between the two, controlling replication costs while enhancing availability through structured redundancy, thereby establishing a new compromise between data availability and cost efficiency.

The Redstuff created by Walrus is a key technology for reducing node redundancy, originating from Reed-Solomon(RS) coding. RS coding is a very traditional erasure code algorithm, and erasure code is a technology that allows for the doubling of a dataset by adding redundant fragments(erasure code), which can be used to reconstruct the original data. From CD-ROMs to satellite communications to QR codes, it is frequently used in daily life.

Erasure codes allow users to take a block, for example, 1MB in size, and then "expand" it to 2MB in size, where the additional 1MB is special data known as erasure code. If any byte in the block is lost, users can easily recover those bytes through the code. Even if up to 1MB of the block is lost, you can recover the entire block. The same technique allows computers to read all the data on a CD-ROM, even if it has been damaged.

The most commonly used is RS coding. The implementation method is to start from k information blocks, construct the related polynomial, and evaluate it at different x coordinates to obtain the encoded blocks. Using RS erasure codes, the probability of randomly sampling large amounts of lost data is very low.

For example: a file is divided into 6 data blocks and 4 parity blocks, totaling 10 pieces. As long as any 6 pieces are retained, the original data can be completely restored.

Advantages: Strong fault tolerance, widely used in CD/DVD, fault-tolerant hard disk arrays (RAID), and cloud storage systems ( such as Azure Storage, Facebook F4).

Disadvantages: Decoding calculations are complex and the overhead is high; it is not suitable for scenarios with frequent data changes. Therefore, it is usually used for data recovery and scheduling in centralized off-chain environments.

Under the Decentralization architecture, Storj and Sia have adjusted traditional RS coding to meet the actual needs of distributed networks. Walrus has also proposed its own variant - the RedStuff coding algorithm - based on this, to achieve lower costs and more flexible redundancy storage mechanisms.

What is the biggest feature of Redstuff? By improving the erasure coding algorithm, Walrus can quickly and robustly encode unstructured data blocks into smaller shards, which are distributed and stored in a network of storage nodes. Even if up to two-thirds of the shards are lost, the original data block can be quickly reconstructed using partial shards. This is possible while maintaining a replication factor of only 4 to 5 times.

Therefore, it is reasonable to define Walrus as a lightweight redundancy and recovery protocol redesigned around a Decentralization scenario. Compared to traditional erasure codes ( such as Reed-Solomon ), RedSt