POIFIER. A user-friendly tool for verification of indexing data consistency

We are excited to announce POIFIER !

Developed jointly by Grassets Tech and Ryabina, POIFIER is The Graph infrastructure project made by Indexers for Indexers on a mission to help with verification of indexing data consistency.

Long story — short

Indexer is a node operator who provides indexing and query processing services for The Graph protocol consumers.

As the proof of data indexing Indexer submits POI (Proof Of Indexing) to the protocol and gets indexing rewards.

Submitted POI for a block is a digest for all entity store transactions for a specific subgraph deployment up to and including that block.

Since fair Indexer strives to provide service by delivering consistent and up-to-date data, the main challenge is verification of indexing data.

Data verification can be achieved by comparing the current POI generated by Indexer for a particular subgraph and Ethereum block to POIs generated by other Indexers if they would generate POI for the same subgraph and Ethereum block using referenced indexer ID.

We aim to build an end-to-end solution to implement this idea which comprises client and server (GUI) applications to help Indexer with verification of their indexing data.

Problem statement

For XYZ reason indexer’s node may experience an issue when node would produce “invalid” POI while no visible issue detected by node operator.

Invalid POI is a symptom of indexing data inconsistency and would lead to a case when the indexer would serve a query request with “wrong” data.

Which POI is valid

So far there is very little visibility and no tool to verify whether generated POI is valid or not, but it was supposed by design that POI generated by any indexer having referenced indexer ID for a particular subgraph and block would match.

There are could be multiple approaches, but we would focus on next 2:

Centralized, aka Oracle: where all POIs generated by indexers would test against POIs generated by single Oracle
Decentralized, aka consensus: where indexer’s POI would test against referenced POI which is being selected as a majority of submitted POIs for given subgraph/epoch/block

Solution

POIFIER is a client-server tool with deep analytics which aims to implement Decentralized approach.

On a regular basis Indexers would submit POIs generated for all subgraphs being indexed by their graph-node for a range of Epochs and Ethereum Blocks using referenced indexer ID (0x0…0)
POIfier-server would select POI based on major number of appearance for each subgraph/block/epoch and name it as a “Valid POI”
POIs would be assessed against “Valid POI”

Benefits

For Indexer:

On early stage to detect, identify and fix single indexer issue
On early stage to detect, identify and flag developer on multiple indexer issue

For Graph Developers:

On early stage to detect, identify and fix indexing issues
To research and investigate multiple indexer / cross Ethereum nodes issue

For Subgraph Developers:

Add visibility into indexing data consistency and be able to identify potential issues with subgraphs.
Improve protocol indexing data consistency

Design

POIFIER infrastructure consists of:

POIfier-client — python3 client which is deployed within Indexer infrastructure, queries indexer’s graph-node against subgraphs and POI per epoch/block and uploads report on poifier-server on regular basis
PPOIfier-server — user friendly GUI with analytics data, where registered indexer is able to submit to and analyse POIs.

How to use

Step by step guide

1: Get API Token

Go to https://poifier.io, connect with Metamask (Metamask account must be linked to the address of the indexer, i.e. it must be the operator or the beneficiary or the indexer itself)