You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug] graphman restore target deployment is dropped by the unused-deployment job while the restore is still running #6752
A long-running graphman restore (more than 6 h) gets its target deployment removed while it is still importing data. The automatic "Record and remove unused deployments" job records the new site as unused. Once GRAPH_REMOVE_UNUSED_INTERVAL (default 360 min) has passed, the job drops the deployment's schema, and the restore fails.
What happened
Version: graph-node / graphman 0.45.0 (the same code path is on current master).
We restored a large deployment into a different shard while another copy of the same deployment hash was active:
The restore created the new site.
About an hour later, the site was recorded in unused_deployments while the restore was still importing data.
Exactly 6 h after unused_at, the site was removed. The restore then failed on its next insert with ERROR: relation "sgdNNN.<table>" does not exist. unused_deployments shows entity_count = 0 for the removed site.
Cause
SubgraphStore::restore (store/postgres/src/subgraph_store.rs) does these steps in order:
create_site(...) creates the site, but no assignment yet.
store.restore(site, dir, ...) imports the data, which can take hours for large subgraphs.
create_subgraph_version(name, &site, node, ...) creates the assignment only at this point.
During step 2 the site meets every criterion in detect_unused_deployments (store/postgres/src/primary.rs):
It is not assigned to a node.
It is not the current or pending version of a subgraph (and it is not active when another copy is active).
It is not the source of an active copy.
As a result, the unused job (which runs periodically from store/postgres/src/jobs.rs) records the site. When the interval has passed, remove_deployment drops it from under the running restore.
Expected behavior
A deployment that a restore is still writing to should never be considered unused.
Possible fix
Record in the primary that a restore is in progress, the same way active_copies protects copy sources: for example a row in a dedicated table, or a flag on deployment_schemas. Write that marker in the same transaction as create_site, exclude marked sites in detect_unused_deployments (and re-check the marker in remove_deployment), and clear it once create_subgraph_version has run. If a restore is interrupted, its site is left marked, so graphman should be able to list and clear stale markers.
Assigning the site early with a paused assignment does not work as a fix: other processes, such as indexer-agent, routinely resume paused deployments. The node would then start indexing a half-restored deployment.
Workaround
Once the restore has created the schema, assign the new site to a placeholder node (use the namespace, because the hash is ambiguous when another copy exists):
graphman reassign sgdNNN placeholder_node
remove_deployment re-checks the assignment before dropping a site, and create_subgraph_version keeps the existing assignment. After the restore finishes, reassign the deployment to a real node. Raising GRAPH_REMOVE_UNUSED_INTERVAL above the expected restore duration also works.
Activity