Astproxy: pending AMI command callbacks are never released on reconnection → nethcti-server hangs at 100% CPU #8273

Description

Steps to reproduce

• On a NethVoice with queues and some call traffic (AMI events flowing to nethcti-server), find the node process of nethcti-server
• Freeze it for a couple of seconds, so it stops reading the AMI socket: kill -STOP <pid>; sleep 2; kill -CONT <pid>
• Asterisk closes the AMI session of the proxycti user (its writetimeout is 100 ms) and astproxy reconnects: the journal shows [astproxy] Error: write ECONNRESET, asterisk socket close and asterisk connected
• Repeat: every reconnection leaves in memory the callbacks of the commands that were in flight on the closed socket

The same happens without any manual action on busy installations: at peak time Asterisk logs Taskprocessor 'stasis/m:manager:core-...' queue reached 3000 scheduled tasks, nethcti-server cannot drain the AMI socket within 100 ms and Asterisk keeps dropping the session (observed up to ~370 reconnections per hour).

Expected behavior

Commands sent on an AMI connection that is closed are discarded when astproxy reconnects. nethcti-server memory stays stable and the API keeps answering after any number of reconnections.

Actual behavior

After a burst of reconnections nethcti-server hangs:

• the main thread runs at 100% CPU and ignores SIGTERM until systemd kills it
• the middleware logs [PROXY][V1] dial tcp 127.0.0.1:<NETHCTI_SERVER_API_PORT>: i/o timeout and every CTI login fails, the CTI is unusable
• the node process grows to more than 1 GB of RSS plus swap
• the log goes silent; only a restart recovers it

Observed twice on the same production installation (NethVoice 1.7.9, then on a 1.8.0 build), each time a few hours after a morning with hundreds of AMI reconnections.

Root cause

Every command plugin in lib/plugins_command_13 stores the callback before sending the action and deletes it only when the final event of that ActionID arrives, e.g. queueDetails.js:

map[act.ActionID] = cb;
am.send(act);
...
if (data.event === 'QueueStatusComplete') {
  delete list[data.actionid];
  delete map[data.actionid];
}

There is no timeout and no cleanup when the AMI connection is closed. asterisk-ami creates a new socket on every reconnection and never resends pending actions, so these responses can never arrive: the entries stay in map (and list / ref) forever. All 63 command plugins follow the same pattern; queueDetails grows fastest, because proxy_logic_13 polls all queues periodically.

On the hung process the map of queueDetails.js held 8,419,805 pending callbacks, read from the process memory. That is above the V8 dictionary enumeration index limit (23 bits, 8,388,607, property-details.h): from that point every insert renumbers and sorts all the keys (NameDictionary::Add → NextEnumerationIndex → std::__introsort_loop<EnumIndexComparator>, where the CPU was found), so the event loop processes one operation every few seconds.

The trigger is the writetimeout=100 of the proxycti AMI user, set in ns8-nethvoice/freepbx/initdb.d/initdb.php: a 100 ms stall of the node event loop is enough for Asterisk to drop the session.

Suggested fix

• astproxy: every command plugin exports a reset() that empties its per-ActionID state (map, plus list / ref where present), and the connection-connect handler calls it on all plugins, logging how many pending commands were discarded. When connection-connect fires, the previous socket is gone and nothing was resent, so no valid response can be lost; the discarded callbacks are not invoked, as they are not today.
• To evaluate separately: raise the writetimeout of the proxycti AMI user (e.g. 5000 ms, like the asterisk user) to reduce the reconnections at peak time.

Components

• @nethesis/astproxy 0.1.2 (also present in master, 0.2.1)
• nethcti-server (branch ns8, shipped in ghcr.io/nethesis/nethvoice-cti-server)
• NethVoice module 1.7.9 / 1.8.0 on NS8