Repository navigation
[Bug] Reassigning subgraph to non-existent indexer doesn't stop indexing #5011
Description
Bug report
It seems that reassigning a subgraph to a non-existent indexer does not stop indexing on 0.33.0. I have a subgraph that's erroring out and I want to reassign it to an indexer that doesn't exist to stop indexing. I know there's a new pause command but that doesn't have a JSON HTTP API so I can't build automation around it.
I just tried reassigning with the following command on the command line. I see that the removal event was received by the graph node indexer, but the indexers continue to index
❯❯❯ kubectl exec --namespace default -it deploy/unicrypt-indexer-0-graph-node -- graphman reassign QmZ1Qt5nxLWDhT6D6L8FyyEHMGukCiP1g6HLuxPjt43UoL paused2
reassigning QmZ1Qt5nxLWDhT6D6L8FyyEHMGukCiP1g6HLuxPjt43UoL[1264] to paused2 (was paused)
warning: this is the only deployment assigned to paused2
are you sure it is spelled correctly?
Relevant log output
unicrypt-indexer-0-graph-node-75ff4dbbff-xr9cc graph-node Nov 22 10:06:02.990 DEBG Received assignment event: Remove { deployment: DeploymentLocator { id: DeploymentId(1264), hash: DeploymentHash("QmZ1Qt5nxLWDhT6D6L8FyyEHMGukCiP1g6HLuxPjt43UoL") }, node_id: NodeId("indexer_0") }, component: SubgraphRegistrar
unicrypt-indexer-1-graph-node-7dd9dbc6c8-bw4wr graph-node Nov 22 10:06:02.991 DEBG Received assignment event: Remove { deployment: DeploymentLocator { id: DeploymentId(1264), hash: DeploymentHash("QmZ1Qt5nxLWDhT6D6L8FyyEHMGukCiP1g6HLuxPjt43UoL") }, node_id: NodeId("indexer_1") }, component: SubgraphRegistrar
unicrypt-indexer-0-graph-node-75ff4dbbff-xr9cc graph-node Nov 22 10:06:23.540 DEBG Starting or restarting subgraph, sgd: 1264, subgraph_id: QmZ1Qt5nxLWDhT6D6L8FyyEHMGukCiP1g6HLuxPjt43UoL, component: SubgraphInstanceManager
unicrypt-indexer-0-graph-node-75ff4dbbff-xr9cc graph-node Nov 22 10:06:23.540 DEBG Starting block stream, sgd: 1264, subgraph_id: QmZ1Qt5nxLWDhT6D6L8FyyEHMGukCiP1g6HLuxPjt43UoL, component: SubgraphInstanceManager
unicrypt-indexer-0-graph-node-75ff4dbbff-xr9cc graph-node Nov 22 10:06:23.540 INFO Blockstream disconnected, connecting, provider_err_count: 0, cursor: TtiCG_7MSiQkWj2ONQIgHKWwLpc_DFttUAriIhBJj4-j9yDGjJqhBmB3YRXTk6j3iRK5Qwn639_LHSsopsYBuYC5l-xlvyBrEnN-k4G4_r3oKaGkawhJculkWuLfMN7RWj7SYw3ze7kK5IDgaPCIYEU0Y8UvJGWwjjgEpddcI_UQuXU8lm-pdZ7XgajCpYMT_rFxTLahnX2mDjIsfhkIP56BNfSat21yYw==, subgraph: QmZ1Qt5nxLWDhT6D6L8FyyEHMGukCiP1g6HLuxPjt43UoL, start_block: 12517819, endpoint_uri: streamingfast-firehose-bsc, provider: streamingfast-firehose-bsc, deployment: QmZ1Qt5nxLWDhT6D6L8FyyEHMGukCiP1g6HLuxPjt43UoL, sgd: 1264, subgraph_id: QmZ1Qt5nxLWDhT6D6L8FyyEHMGukCiP1g6HLuxPjt43UoL, component: FirehoseBlockStreamIPFS hash
QmZ1Qt5nxLWDhT6D6L8FyyEHMGukCiP1g6HLuxPjt43UoL
Subgraph name or link to explorer
No response
Some information to help us out
- Tick this box if this bug is caused by a regression found in the latest release.
- Tick this box if this bug is specific to the hosted service.
- I have searched the issue tracker to make sure this issue is not a duplicate.
OS information
Linux
Activity
paymog commentedon Nov 22, 2023
It seems that even using
graphman pause QmZ1Qt5nxLWDhT6D6L8FyyEHMGukCiP1g6HLuxPjt43UoLdoes not stop the subgraph from indexingpaymog commentedon Nov 22, 2023
turns out restarting both indexers in my cluster forced the subgraph to stop indexing. Something about the messaging between graph nodes to pause indexing was broken.
fordN commentedon Apr 8, 2024
Closing as it seemed to be a one-off issue.