The Importance of Blockchain Data and the Rise of Indexers

Data is the core of Blockchain technology and the foundation for developing decentralized applications (dApp). While most discussions currently focus on data availability (DA), data accessibility is equally important but often overlooked.

In the era of modular Blockchain, DA solutions have become an indispensable part. They ensure that all participants have access to transaction data, enabling real-time verification and maintaining network integrity. However, the DA layer resembles a bulletin board rather than a database, as data is not stored long-term.

In contrast, data accessibility focuses on the ability to retrieve historical data, which is crucial for developing dApps and Blockchain analysis. Although less discussed, data accessibility is equally important as data availability. Both play different but complementary roles in the Blockchain ecosystem, and a comprehensive data management approach must address both issues simultaneously to support robust and efficient Blockchain applications.

Traditional Methods for Blockchain Data Retrieval

The emergence of Blockchain has completely transformed infrastructure, driving the creation of dApps in fields such as gaming, finance, and social networks. However, building these dApps requires access to a large amount of Blockchain data, which is both difficult and expensive.

For developers, one option is to host and run their own archival RPC nodes. These nodes store all historical Blockchain data, allowing for complete access. However, the maintenance costs are high, and the query capabilities are limited. Running cheaper nodes is another option, but the data retrieval capabilities are limited, which may affect the operation of dApps.

Using commercial RPC node providers is another approach. These providers are responsible for node costs and management, providing data through RPC endpoints. Public RPC endpoints are free but have rate limits, which may affect user experience. Private RPC endpoints perform better, but even simple data retrieval requires a lot of communication, leading to inefficiency and difficulty in scaling.

Blockchain Indexer: A Better Solution

Blockchain indexers play a critical role in organizing chain data and sending it to databases for querying, hence they are referred to as "the Google of Blockchain." They index blockchain data and make it available through a SQL-like query language. Indexers provide a unified query interface, significantly simplifying the data retrieval process.

Different types of indexers optimize data retrieval:

1. Full Node Indexer: Extracts data directly from the full Blockchain node, ensuring completeness and accuracy, but requires significant storage and processing power.

2. Lightweight Indexer: Retrieves specific data from full nodes as needed, reducing storage requirements but may increase query times.

3. Specialized Indexers: Optimized for specific data types or Blockchains, such as NFT data or DeFi transactions.

4. Aggregated Indexer: Extracts data from multiple blockchains and sources, including off-chain information, providing a unified query interface, suitable for multi-chain dApps.

The indexer protocol deploys multiple indexers to efficiently index and quickly query large amounts of data. They also allow for complex queries, data filtering, and post-analysis. Some indexers can aggregate data from multiple sources, avoiding the need to deploy multiple APIs in multi-chain dApps. Spread across multiple nodes, indexers provide better security and performance.

Overall, compared to RPC node providers, indexers improve the efficiency and reliability of data retrieval while reducing deployment costs. This makes the Blockchain indexer protocol the preferred choice for dApp developers.

Application Scenarios of Indexers

Building dApps requires retrieving and reading blockchain data to operate services. This includes various dApps such as DeFi, NFT platforms, games, and even social networks, as they need to read data first to execute other transactions.

DeFi

DeFi protocols require various information to provide users with specific prices, rates, and fees. Automatic Market Maker (AMM) needs price and liquidity information from liquidity pools to calculate swap rates, while lending protocols need utilization rates to determine lending rates and the debt ratio for liquidation. This information is crucial for the dApp before calculating the rates executed by users.

Game

GameFi requires rapid indexing and access to data to ensure a smooth gaming experience for users. Only through fast data retrieval and execution can Web3 games compete with Web2 games in performance, thereby attracting more users. These games require data such as land ownership, in-game token balances, and in-game operations. Using an indexer can better ensure stable data flow and uptime, guaranteeing a perfect gaming experience.

NFT

NFT markets and lending platforms need to index various information, such as NFT metadata, ownership and transfer data, royalty information, etc. Quickly indexing this data can avoid browsing each NFT individually to find ownership or attribute data.

Whether it's DeFi AMM that requires price and liquidity information, or SocialFi applications that need to update new user posts, quickly retrieving data is crucial for the normal operation of dApps. With the help of indexers, they can efficiently and accurately retrieve data, providing a smooth user experience.

Analysis

The indexer provides a method to extract specific data from the raw Blockchain data (, including smart contract events in each Block ). This offers an opportunity for more specific data analysis, leading to comprehensive insights.

For example, perpetual trading protocols can identify which tokens have high trading volumes and generate fees, thus deciding whether to list them as perpetual contracts. DEX developers can create dashboards for their products to gain insights into which liquidity pools offer the highest returns or strongest liquidity. They can also create public dashboards that allow developers to flexibly query any type of data and display it in charts.

As there are multiple blockchain indexers available, understanding the differences between them is crucial for developers to choose the indexer that best fits their needs.

Overview of the Main Blockchain Indexer

The Graph

The Graph is the first indexing protocol launched on Ethereum, allowing for easy querying of previously hard-to-access transaction data. It uses subgraph definitions and filters to collect subsets of data from the Blockchain, such as all transactions related to a specific funding pool.

Indexers stake the native token GRT for indexing and querying services, and delegators can choose to stake tokens. Curators can access high-quality subgraphs to help indexers determine which subgraphs to index to earn the best query fees. The Graph is gradually transitioning towards greater decentralization and will eventually stop hosting services, requiring subgraphs to upgrade to its network.

Its infrastructure brings the average cost per million queries to $40, far lower than self-hosted nodes. By using file data sources, it also supports parallel indexing of both on-chain and off-chain data, enabling efficient data retrieval.

The indexing rewards of The Graph have steadily increased over the past few quarters, partly due to the increase in query volume and also thanks to the rise in token prices. They plan to integrate AI-assisted queries in the future.

Subsquid

Subsquid is a peer-to-peer, horizontally scalable decentralized data lake that efficiently aggregates large amounts of on-chain and off-chain data, protected by zero-knowledge proofs. As a decentralized worker network, each node is responsible for storing a specific subset of Block data, accelerating the retrieval process by quickly identifying the nodes that hold the required data.

Subsquid supports real-time indexing, allowing indexing before the block is finalized. It also supports storing data in formats chosen by developers, making it easier to analyze using various tools. Additionally, subgraphs can be deployed on the Subsquid network without code.

Although still in the testnet phase, Subsquid has achieved impressive results, with over 80,000 testnet users, more than 60,000 deployed indexers, and over 20,000 verified developers on the network. Subsquid recently launched its data lake mainnet.

In addition to indexing, the Subsquid Network data lake can also replace RPC in use cases such as analytics, ZK/TEE co-processors, AI agents, and oracles.

SubQuery

SubQuery is a decentralized middleware infrastructure network that provides RPC and indexing data services. It originally supported Polkadot and Substrate networks and has now expanded to over 200 chains. Its operation is similar to The Graph using index proofs, but it introduces consumers to submit purchase orders to ensure indexer revenue.

SubQuery will introduce sharded data nodes to optimize query efficiency while achieving a greater degree of decentralization. Users can choose to pay a computation fee of approximately 1 SQT token for every 1000 requests, or set a custom fee for the indexer through the protocol.

Although SubQuery launched its token this year, the issuance rewards for nodes and delegators have also shown a month-on-month increase in USD value, reflecting the growing number of query services offered on its platform. Since the TGE, the total amount of staked SQT has increased from 6 million to 125 million, highlighting the growth in network participation.

Covalent

Covalent is a decentralized indexing network, created by Block sample producers (BSP) nodes that generate copies of blockchain data and publish proof on the Covalent L1 chain. This data is then filtered by Block result producers (BRP) nodes based on the rules to meet the required criteria.

By using a unified API, developers can easily extract Blockchain data in a consistent format without writing complex queries. These pre-configured datasets can be extracted from network operators using CQT tokens settled on Moonbeam.

The rewards of Covalent show an overall upward trend from the first quarter of 2023 to the first quarter of 2024, partly due to the increase in the price of CQT tokens.

Factors to Consider When Choosing an Indexer

Data Customizability

Some indexers ( such as Covalent ) provide standard pre-configured datasets only through APIs. While they are fast, they lack flexibility. Using an indexer framework allows for more customized data processing to meet specific application needs.

Security

Index data must be secure; otherwise, dApps built on these indexers are also vulnerable to attacks. All indexers ensure security through staking tokens, but some solutions also use additional proof mechanisms.

Subsquid provides optimistic and zero-knowledge proof options, Covalent releases proofs containing block hashes. The Graph offers a dispute challenge period for indexer queries, and SubQuery generates Merkle Mountain proofs for each block.

Speed and Scalability

As blockchain growth and transaction volume increase, indexing large amounts of data has become more cumbersome. The indexer protocol introduces solutions to address these demands.

Subsquid horizontally scales by adding more nodes. The Graph provides parallel streaming data to speed up synchronization, while SubQuery introduces node sharding to accelerate the synchronization process.

Supported Networks

Although Ethereum remains the main activity hub, other blockchains are becoming increasingly popular. Providing services for chains not supported by other indexers can gain more market share. Indexing data-intensive networks ( such as Solana ) is not easy, and currently only Subsquid has successfully provided indexing support for it.

Conclusion

Although indexers are widely used in dApp development, their potential remains vast, especially in terms of integration with AI. As AI becomes more prevalent in Web2 and Web3, its improvement depends on accessing relevant data to train models and develop AI agents. Ensuring data integrity is crucial for AI applications, as it prevents models from being influenced by biased or inaccurate information.

In the field of indexing solutions, Subsquid has made significant progress in performance and user metrics. Users have started experimenting with building AI agents using Subsquid, showcasing the platform's versatility and potential. Additionally, tools like AutoAgora assist indexers in using AI to provide dynamic pricing for query services on The Graph, while SubQuery supports multiple AI networks for transparent data indexing.

The integration of AI and indexers is expected to enhance data accessibility and availability within the blockchain ecosystem. By leveraging AI technology, indexers can provide more efficient and accurate data retrieval, enabling developers to build more complex dApps and analytical tools. As AI and indexers continue to evolve together, we remain optimistic about the future of data indexing and its role in shaping the decentralized digital landscape.