Repository navigation
Indexing stucks until restart #4146
Description
Sometimes one of the subgraphs stops indexing without errors, while others continue syncing. The only way to fix it is to restart the whole indexer. Every time it is a different subgraph. It doesn't matter if it is self-written or forked from a hosted service using an IPFS hash.
I'm deploying the graph-node to Kubernetes, to the same cluster with the Ethereum node (erigon) and Postgresql. Version is 0.28.2 with GRAPH_KILL_IF_UNRESPONSIVE set to true.
The logs are clear. The indexer stops producing records related to the stuck subgraph_id at all.
I have not found related issues, so I kindly ask for tips on diagnosing a problem in my setup.
Screenshots from Grafana for subgraph Qmca1arSe9U94sHVpzzdUKizNyqFddsXnBZYqkALmSdSRE https://thegraph.com/hosted-service/subgraph/indexed-finance/indexed as an example.
Activity
azf20 commentedon Nov 7, 2022
hey @serjvanilla thanks for the report - are you able to run your Graph Node with
GRAPH_LOG=traceto see if you get further details on the affected subgraph?serjvanilla commentedon Nov 8, 2022
Indexer redeployed with
GRAPH_LOG=trace. Still, no stops since, but I'm continuously watching it. I will try to find more info in that logs as soon as any subgraph stops again.azf20 commentedon Nov 29, 2022
Hi @serjvanilla closing this for now, but please do re-open if you saw stops as you described above!