Module restart kills running bash tasks, but AFT reports them as 'process fate unknown'

Problem

In subc mode, a module restart kills every running bash task, but after the restart AFT reports these tasks as fate_unknown with the text "daemon restarted, process fate unknown, last output at …". The agent cannot tell that the command was killed. It can wait for a result that will not come, or read a partial log as if the command is still running.

Also, the kill is a plain SIGKILL from outside AFT. AFT's own kill path (SIGTERM, 2 s grace, then SIGKILL, bash_background/process.rs:13) does not run, so trap … EXIT cleanup in the command does not run.

Evidence (aft 0.62.0 module, one supervisor restart)

The daemon started the module in its own cgroup, …/subc-modules/m-aft_<id>. Each bash task that AFT spawns is in the same cgroup (/proc/self/cgroup of a tool shell shows the module's cgroup). The daemon uses cgroup.kill on module cgroups (the string is in the daemon binary; I did not read the daemon source).

At the restart:

21:02:39.757Z  daemon: module drain began … reason=Restart
21:02:41.249Z  aft: subc module stopped at the daemon's request; exiting 0
21:02:42.856Z  daemon: module registered module_id=aft
21:02:45-46Z   aft: 4 tasks marked fate_unknown

All four tasks that were running stopped at the module exit:

No task has an exit marker file. The wrapper did not get to printf "%s" "$code" >&"$exit_fd".
One task wrote to its stderr file until 21:02:42.26Z, 1 s after the module exit, and then stopped.
No process of any of the four process groups is alive.

Two of the four fate_unknown notices were delivered (completion_delivered=true). The other two were not delivered when I checked, because their sessions had not bound again.

Code (main `a6a3a8006`)

crates/aft/src/bash_background/registry.rs:7082-7083: on replay, a task with a child identity, no exit marker and no live process becomes FateUnknown.
registry.rs:14514 restart_fate_unknown_reason writes "daemon restarted, process fate unknown".
The subc exit phases (drain_done, index_flush, lsp_shutdown, runtime_shutdown) do nothing to running bash tasks. BgTaskRegistry::shutdown(), which kills all tasks, has callers only in tests.

Questions

When the daemon drains the module with reason=Restart or Stop, can AFT end its running tasks itself in the exit phase (SIGTERM, grace, SIGKILL, the same as a timeout kill)? Then the command's EXIT traps run, and AFT can record a definite result such as "killed: aft module restarted () at ", with the output tail.
If not, can the replay text say what is very possible: the module restarted and the daemon killed the task's cgroup? "Fate unknown" makes agents wait.
Do you want long tasks to continue after a module restart (for example, with a task cgroup that is not in the module cgroup)? That also needs a daemon change, so it can be a different issue.

More information from the daemon side, for question 3.

After the module exits, the daemon ends what stays in the module cgroup. A proposed daemon change adds SIGTERM and a short grace before cgroup.kill. This lets EXIT traps run, but it still ends the tasks. A module cgroup ends with its module.

So, for a task to continue after a module restart, AFT must start it outside the module cgroup. bash.linux_scope: true already starts non-PTY tool shells with systemd-run --user --scope. A transient scope is in the user manager's tree, not in the module cgroup, so the daemon's cleanup does not reach it. The other parts seem to be in place:

The wrapper writes the exit code to io/exit in the task directory, which is a file, not a pipe to the module process.
On replay, a task with a live process becomes Running again, and a task with an exit marker gets its real result (registry.rs, replay of Starting/Running tasks).

I did not test this through a module restart. If it works, two changes seem enough: document that linux_scope lets tasks continue after a module restart, and consider it as the default when systemd-run and the user manager are available. Tasks that do not use a scope (PTY, native sandbox, linux_scope off) still need questions 1 and 2.

Thanks, this is a clear report. On Linux, when the AFT module restarts, the Subconscious daemon kills the module's whole cgroup, so background commands started through AFT die with it. On macOS they survive by design. The daemon does this on purpose, because it can't remove a cgroup that still has processes in it.

We'd like these commands to survive a module restart on Linux too, by running them in a separate cgroup. Would you be interested in working on that? Please post a short design on this issue first. We'll review it before any code is written. It should cover:

Which cgroup the commands run in, who creates it, and who holds the rights to it. It sits next to the module's own cgroup, so the Subconscious daemon is part of this.
How the existing controls still reach the commands: stopping a command on request, the memory cap, and ending a session's commands when its scope ends.
How AFT finds the running commands again after it restarts and keeps reporting their output and exit status.
What ends those commands when AFT stops for good, so nothing is left running forever.
What happens on Linux systems without systemd or without cgroup v2.

Once the design is agreed, a PR is welcome.

Design: bash tasks continue after an AFT module restart on Linux

Yes, I am interested. This is the design for review. No code is written yet.

Summary

Each non-PTY task runs in its own transient systemd user scope, aft-task-<task_id>.scope in aft.slice. The user's systemd manager creates and owns the scope. The scope is not below the daemon's service cgroup, so the daemon's module cleanup does not reach it, and the daemon needs no change. This is the current bash.linux_scope with five changes:

Repair the argument expansion defect (bash.linux_scope: every non-PTY command fails with exit 2 on systemd 254+ (systemd-run expands ${…} and $$ in the wrapper) #1134). At the moment linux_scope: true makes all commands stop with exit code 2 on systemd 254 and later.
Give the scope a name and a slice, a deadline backstop, and a stop grace.
After the process-group kill, also end the scope's cgroup.
Put the memory cap on the same scope.
Use the scope by default when systemd is available.

What I tested (systemd 262, aft 0.62.0 module under the daemon)

From a tool shell inside the AFT module cgroup, systemd-run --user --scope … put the command in its own scope below the user manager's app.slice, outside the daemon's service cgroup. -p MemoryMax=64M set memory.max to 67108864 in that scope.
Simulation: a "module" process in scope A started a "task" in scope B. I wrote 1 to scope A's cgroup.kill, as the daemon does. The module stopped. The task continued, and after it completed it wrote its exit file.
A scope with RuntimeMaxSec=3 got SIGTERM after about 3 s and the unit stopped. The process that called systemd-run had exited earlier, and this did not change the result.
systemctl --user set-property --runtime <scope> RuntimeMaxSec=… fails on a running scope: "Cannot set property RuntimeMaxUSec". This affects point 2 below.
With --slice=aft-tasks.slice, systemd makes the hierarchy aft.slice/aft-tasks.slice, so I propose the plain aft.slice.

1. Which cgroup, who creates it, who holds the rights

The spawn argv becomes:

systemd-run --user --scope --collect --quiet --expand-environment=no \
  --unit=aft-task-<task_id> --slice=aft.slice \
  -p RuntimeMaxSec=<backstop> -p TimeoutStopSec=2s \
  [-p MemoryMax=<bytes> -p MemorySwapMax=0] \
  /bin/sh -c <wrapper> …

systemd-run --scope asks the user manager over D-Bus to create the scope with its own PID, then runs exec on the wrapper. Thus the command is in the scope before any user code runs.
The cgroup is <user manager>/aft.slice/aft-task-<task_id>.scope. The user manager owns it. AFT runs as the same user, and the user manager's subtree is delegated to that user, so AFT can read the scope's files (cgroup.events, memory.events) and write cgroup.kill. With this design, AFT does not write to the daemon's cgroups.
The daemon is not involved. Its cleanup after a module exit (cleanup_reaped_cgroup in supervise.rs) and kill_module act only on the module's own cgroup and its descendants.

Alternatives that I did not select:

A cgroup that AFT makes next to its module cgroup in subc-modules/. That subtree belongs to the daemon. No controllers are enabled there (cgroup.subtree_control is empty on the live host), so a memory cap is not possible. Also, nothing removes empty leftovers, and the daemon gives no promise about unknown children.
A daemon API that gives each module a task cgroup that stays across restarts. This is the cleanest governance, and it works without systemd when the daemon can write cgroup v2. But it needs a protocol change and a daemon release. It also gives no memory cap unless the daemon enables the memory controller. I suggest it only as a later fallback for hosts with cgroup v2 but no systemd user manager, if users ask for it.

2. How the existing controls still reach the commands

bash_kill, timeout, end of a session scope, root removal. The first step does not change: SIGTERM to the process group, TERMINATE_GRACE (2 s), then SIGKILL. New: after this, if the task has a scope, AFT writes 1 to the scope's cgroup.kill. This also ends descendants that left the process group (setsid, double fork), which survive the current kill.
Memory cap (bash.memory_limit_mib). MemoryMax and MemorySwapMax=0 go on the task scope when systemd creates it, and AFT writes memory.oom.group=1 as it does now. For these tasks this replaces the bootstrap helper and the cgroup.procs attach. AFT reads OOM events from the scope's memory.events. AFT must not use the first attempt in memory_limit.rs (delegated), which makes a child of AFT's own cgroup, for a task that must continue after a restart, because that child is in the module's kill domain.
Deadline backstop. RuntimeMaxSec ends the scope if AFT does not end the task first. For an explicit timeout (HardKill::After) it is the timeout plus 60 s, so that AFT's own kill runs first and reports TimedOut. A delegated worker can extend a HardKill::Default task (renew_hard_kill), but systemd cannot change RuntimeMaxSec on a running scope. For these tasks I propose a fixed ceiling (24 h, configurable). If you prefer, the wrapper can read a deadline file that AFT updates instead. This is the first question for review.
TimeoutStopSec=2s. When systemd stops the scope (backstop, slice stop), the processes get SIGTERM, 2 s of grace, then SIGKILL. This is the same as AFT's own kill, so EXIT traps run.

3. How AFT finds the commands after it restarts

Most of this is in the code at this time. It is the path that works on macOS:

The task record has child_pid, pgid and the child's start time (persistence.rs). Replay (registry.rs, around line 7385) reads io/exit. If there is no marker, it checks the PID against its start time. A live non-PTY task becomes Running, and AFT adopts it through exit_observer.rs (pidfd).
Output goes to files in the task directory. The wrapper writes the exit code through an fd that points to the io/exit file. No pipe connects the task to the module process.
A task that finished while AFT was down has its marker. Replay gives the correct exit code and delivers the completion one time, not FateUnknown.

New: the unit name comes from the task id, so a new record field is not necessary. At replay, AFT also checks that /proc/<pid>/cgroup ends in aft-task-<task_id>.scope. This is a second identity check after the start time. AFT then keeps the scope path for kill and memory events.

FateUnknown stays only for tasks without a scope (see 5). For those, questions 1 and 2 of this issue continue to apply.

4. What ends the commands when AFT stops for good

The daemon's stop request to a connected module is EOF on its connection (child_roster.rs in the daemon), so the module does not know if the stop is a restart. Thus AFT does not end its tasks when it exits, and these backstops apply:

For each task: RuntimeMaxSec (section 2). No task runs longer than its deadline plus the margin, also if AFT does not start again.
At startup: AFT lists the aft-task-*.scope units in aft.slice. If its own storage has a record for the task id and the record is terminal, AFT stops the scope. AFT does not touch units without a record in its storage, because a different AFT process (for example, a standalone one) can own them.
Operator: systemctl --user stop aft.slice stops all AFT tasks. The docs give this command.

Behavior change: the tasks also continue after a full stop of the daemon service, because they are not in its cgroup. At the moment a service stop ends them. With the backstop, no task continues after its own deadline.

5. Linux without systemd or without cgroup v2

No systemd user manager (no systemd-run, no user bus, some containers): the probe does not find a manager, and AFT starts the task as it does at this time, without a scope.
- If the daemon put AFT in a module cgroup, the task stops with the module, as it does at this time. AFT then reports it clearly (questions 1 and 2 of this issue).
- If the daemon has no placement (it logs "module cgroup placement is disabled"), tasks continue after a module restart at this time, because each task has its own session, as on macOS. Replay adopts them. AFT's own timeout applies again when AFT starts. While AFT is down there is no backstop.
systemd before 254: --expand-environment=no is not available (bash.linux_scope: every non-PTY command fails with exit 2 on systemd 254+ (systemd-run expands ${…} and $$ in the wrapper) #1134). AFT starts the task without a scope, unless it escapes $ in the arguments.
PTY tasks: AFT holds the PTY master. When AFT exits, the terminal closes and the task gets SIGHUP. These tasks cannot continue without a different PTY holder process, which is out of scope. Replay marks them Killed (pty_lost_on_bridge_restart).
Native sandbox: at the moment AFT does not use the scope with the native launcher, because the sandboxed launcher cannot connect to the user manager. Possible subsequent step: run systemd-run before the launcher (systemd-run … <launcher> … /bin/sh …). systemd-run --scope makes its D-Bus call and then runs exec on the launcher. I did not test this.

Configuration

bash.linux_scope default changes from false to true. When the probe does not find a manager, AFT logs one line and starts tasks without a scope. false disables the scope.

Tests

Unit: the argv has --expand-environment=no, --unit, --slice, RuntimeMaxSec, TimeoutStopSec, and MemoryMax when a cap is set.
Linux integration (skip and give the cause when no user manager is available): run AFT in a transient scope as the module cgroup and start a background task. Write cgroup.kill on AFT's scope, then start a new AFT on the same storage. The task replays as Running, then completes with its correct exit code and full output.
Kill: a task forks a child that calls setsid. After bash_kill, the scope cgroup is empty.
Backstop: AFT is gone, and a task with a short timeout stops at RuntimeMaxSec.
Startup cleanup: a terminal record with a live scope gets stopped. A unit without a record stays.
Expansion: a command with ${…} and $$ gives the same output with and without the scope.

Order of PRs

bash.linux_scope: every non-PTY command fails with exit 2 on systemd 254+ (systemd-run expands ${…} and $$ in the wrapper) #1134 (expansion), small and independent.
Scope name, slice, backstop, stop grace, scope kill, replay identity check, startup cleanup.
Memory cap on the task scope.
Default on.
Subsequently, if necessary: scope with the native sandbox, and the daemon task cgroup for hosts without systemd.

Questions for review

For HardKill::Default tasks: a constant backstop ceiling (24 h, configurable), or a deadline file that the wrapper checks?
Do you want the bash result to show when a task runs without a scope, so it will not continue after a restart?
Are the names aft.slice and aft-task-<task_id>.scope satisfactory?

Thanks for the design. We went through it with the Subconscious maintainers, and here is what we decided:

Go ahead and build it as designed: each non-PTY background task in its own systemd user scope in aft.slice, outside the module cgroup.
Daemon mode only. Use the scope only when AFT runs under the Subconscious daemon. In standalone mode nothing kills AFT's cgroup and tasks already start detached, so standalone stays as it is.
On by default wherever systemd is available.
Time limit: commands without their own timeout get a fixed 24-hour ceiling through RuntimeMaxSec, configurable.
No notice when a command can't survive a restart (no systemd). If such a command dies, AFT reports it honestly, as it does today.
A real stop still ends the commands. The daemon already tells AFT why it is draining the module (module.draining with a reason, on_draining in subc-client-rs): restart and reload keep the task scopes running, while disable, or a daemon shutdown with no draining notice, ends them. We'll do the AFT handling of that reason; your part can assume it.
Long term, a daemon-owned task cgroup will replace the scope, and it also works without systemd. Please keep the scope handling behind one seam so it can be swapped later.

The systemd-run argument expansion fix from #1134 is merged, so you can build on current main. Happy to review the PR.

Duplicate of the comment above (posted twice by a retry); please ignore.

The pull request for this design is #1219.

It follows the decisions in this thread: daemon mode only, on by default when a systemd user manager is available, a constant 24-hour ceiling that the user can change, no new notice, and all scope operations in one module. For the drain-reason change, end_all_task_scopes(&self, storage_dir: &Path) stops all task scopes that one storage owns. At this time, no code calls it.

The design-gate check on #1219 fails because this issue does not have the design-approved label. Please add the label if you continue to approve the design. Then the build and test checks can run, with the Windows -D warnings check. I could not build for Windows or macOS here, so that check is the first test of the non-Linux code.

Navigation Menu

Repository navigation

Module restart kills running bash tasks, but AFT reports them as 'process fate unknown' #1109

Problem

Evidence (aft 0.62.0 module, one supervisor restart)

Code (main `a6a3a8006`)

Questions

Design: bash tasks continue after an AFT module restart on Linux

Summary

What I tested (systemd 262, aft 0.62.0 module under the daemon)

1. Which cgroup, who creates it, who holds the rights

2. How the existing controls still reach the commands

3. How AFT finds the commands after it restarts

4. What ends the commands when AFT stops for good

5. Linux without systemd or without cgroup v2

Configuration

Tests

Order of PRs

Questions for review

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Module restart kills running bash tasks, but AFT reports them as 'process fate unknown' #1109

Description

Problem

Evidence (aft 0.62.0 module, one supervisor restart)

Code (main a6a3a8006)

Questions

Activity

iceteaSA commented on Oct 28, 2026

aft-alfonso commented on Oct 28, 2026

iceteaSA commented on Oct 29, 2026

Design: bash tasks continue after an AFT module restart on Linux

Summary

What I tested (systemd 262, aft 0.62.0 module under the daemon)

1. Which cgroup, who creates it, who holds the rights

2. How the existing controls still reach the commands

3. How AFT finds the commands after it restarts

4. What ends the commands when AFT stops for good

5. Linux without systemd or without cgroup v2

Configuration

Tests

Order of PRs

Questions for review

aft-alfonso commented on Oct 29, 2026

aft-alfonso commented on Oct 30, 2026

iceteaSA commented on Oct 30, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Code (main `a6a3a8006`)