Skip to content

NethVoice: Monitor service availability #8093

Open
Backend
1 of 1 issue completed
Open
NethVoice: Monitor service availability#8093
Backend
1 of 1 issue completed
Assignees
andrea-marchionni
Labels
nethvoiceBug or features releted to the NethVoice projecttestingPackages are available from testing repositories
Milestone
NethVoice 1.8

Description

@Amygos
Amygos
opened on Jul 8, 2026

Description

Add the first set of built-in alerts for NethVoice and NethVoice proxy, as part of #8066, so administrators know about problems before users report them.

Each module exports its own metrics and publishes its own alert rules. The node runs the export-metrics command of each module every 5 minutes (NethServer/ns8-core#1365). The metrics app evaluates the rules stored by the module (NethServer/ns8-metrics#91) and sends notifications with its existing portal and email settings.

The full list of alerts, with severity and conditions, is in this comment.

Scope

  • Services: ServiceDown for the NethVoice services (FreePBX/Asterisk, MariaDB, Janus, CTI, Phonebook, Tancredi, Satellite, Reports, SFTP, SAML2/OAuth2 proxies) and the proxy services (kamailio, postgres, redis, rtpengine). Service state is read from systemd, not from podman.
  • Trunks: NethVoiceTrunkUnreachable. NethVoice checks trunks that bypass the proxy; NethVoice proxy checks UDP trunks that go through it. TCP and TLS trunks through the proxy are not checked yet.
  • DNS: NethVoiceDnsFailure and NethVoiceDnsSlow for the NethVoice, CTI and proxy names, checked on the DNS resolver of the node (/etc/resolv.conf).
  • AI: NethVoiceAiKeyInvalid and NethVoiceAiCreditExhausted for the Deepgram and OpenAI keys used by transcription.
  • Alerts are off by default and are turned on per module from the "Alerts" card of the Settings page (set-alerts action). The list of alerts and their thresholds cannot be changed.

Call quality and configurable thresholds belong to the parent issue.

Acceptance criteria

  • With alerts off (default), no rules are published, but metrics are still exported. Turning alerts on publishes the rules; turning them off removes them.
  • Rules are published at module create, update, restore and clone, and removed when the module is removed. Rules of other modules are not touched.
  • ServiceDown fires with reason=stopped when a service is stopped for 10 minutes, or with reason=crash_loop when systemd restarts it at least 3 times in 10 minutes. Disabled services and services never started are not watched.
  • NethVoiceTrunkUnreachable fires when a trunk stays unregistered or its provider does not answer for 10 minutes.
  • NethVoiceDnsFailure fires when a name does not resolve (reason=not_resolved) or the proxy name does not point to the proxy IP (reason=ip_mismatch) for 15 minutes. NethVoiceDnsSlow fires when the resolver is slower than 3 seconds or does not answer for 15 minutes.
  • NethVoiceAiKeyInvalid fires when transcription is on and the provider rejects the API key for 10 minutes. NethVoiceAiCreditExhausted fires when Satellite reports a call refused for lack of credit, and clears after the next successful call.
  • A check that cannot run (for example AI provider unreachable or FreePBX stopped) is reported with nethvoice_exporter_collector_up = 0 (nethvoice_proxy_exporter_collector_up for the proxy) and raises no alert.
  • Alerts show the service or trunk, module instance and node, in English and Italian.
  • Requires core 3.24.0 or later and metrics app 1.7.0 or later. With an older metrics app the rules are ignored.

Verification

  • Unit tests for the metric collectors and rule publishing, and promtool tests for the alert rules, in both repositories.
  • Robot tests in CI for export-metrics, set-alerts and rule publishing.
  • Manual QA on a test cluster: services stopped and in crash loop, trunk failures, DNS not_resolved and ip_mismatch, invalid AI key, and rule cleanup on module removal.

References

Activity

  1. Amygos
    self-assigned this
    on Jul 8, 2026
  2. Amygos
    added
    nethvoiceBug or features releted to the NethVoice project
    on Jul 8, 2026
  3. github-project-automation
    added this to NethVoiceon Jul 8, 2026
  4. Amygos
    added a parent issue on Jul 8, 2026
  5. github-project-automation
    moved this to Todo in NethVoiceon Jul 8, 2026
  6. Amygos
    added the
    Backend
    issue type on Jul 8, 2026
  7. Amygos
    moved this from Todo to In Progress in NethVoiceon Jul 8, 2026
  8. Amygos
    linked a pull request that will close this issuefeat(metrics): add systemd exporter #194on Jul 8, 2026
  9. Amygos
    linked a pull request that will close this issueAdd systemd Prometheus exporter #907on Jul 8, 2026
  10. Amygos
    linked a pull request that will close this issueAdd NethVoice failed unit alerts #78on Jul 8, 2026
  11. Amygos
    added this to the NethVoice 1.7 milestone on Jul 8, 2026
  12. Amygos
    changed the title [-]NethVoice: Monitor failed systemd units[/-] [+]NethVoice: Monitor service availability[/+] on Sep 25, 2026
  13. andrea-marchionni
    modified the milestones: NethVoice 1.7, NethVoice 1.8 on Oct 12, 2026
  14. gsanchietti
    added a sub-issue on Oct 22, 2026
  15. gsanchietti
    assigned
    gsanchietti
    and unassigned
    Amygos
    on Oct 23, 2026

  16. gsanchietti commented on Oct 27, 2026

    @gsanchietti
    gsanchietti
    on Oct 27, 2026
    Member

    Implemented alerts

    Alerts implemented in nethesis/ns8-nethvoice#1067 and nethesis/ns8-nethvoice-proxy#231. They are off by default and are enabled from the "Alerts" card in the Settings page of each module. Checks run every 5 minutes.

    NethVoice (nethesis/ns8-nethvoice#1067)

    Alert Severity Fires when
    ServiceDown critical a NethVoice service is stopped for 10 minutes (reason=stopped) or restarted at least 3 times in 10 minutes (reason=crash_loop). Services: freepbx (Asterisk), mariadb, janus, nethcti-ui, nethcti-server, nethcti-middleware, phonebook, tancredi, satellite, satellite-mqtt, satellite-pgsql, reports-api, reports-redis, reports-ui, sftp, saml2-proxy, oauth2-proxy. Disabled services and services never started are not watched.
    NethVoiceTrunkUnreachable critical for 10 minutes a trunk that bypasses the proxy is not registered (reason=registration_failed), the remote side of a receive trunk is not registered (reason=not_registered), or the provider of an IP trunk that bypasses the proxy does not answer OPTIONS (reason=qualify_failed)
    NethVoiceTrunkNotMonitored warning for 10 minutes an IP or receive trunk has a name Asterisk cannot use as SIP user (a space or another character outside letters, digits and _ . + - ! ~ * ' ( )) and no From User, so its provider is never checked (reason=name_invalid). Fix: rename the trunk without spaces
    NethVoiceDnsFailure warning for 15 minutes the NethVoice, CTI or proxy name does not resolve on the DNS server of the node (reason=not_resolved), the proxy name does not point to the proxy public or private IP (reason=ip_mismatch), or the DNS server does not answer within 5 seconds (reason=no_answer)
    NethVoiceDnsSlow warning for 15 minutes the DNS server of the node takes more than 3 seconds to resolve one of those names; never together with NethVoiceDnsFailure for the same name
    NethVoiceAIKeyInvalid critical with transcription on, Deepgram or OpenAI rejects the API key for 10 minutes
    NethVoiceAICreditExhausted critical Satellite reports that a provider refused a call for lack of credit; clears after the next successful call

    DNS checks use the DNS server configured on the node (/etc/resolv.conf), not public DNS servers. The DNS server answer time is also exported as nethvoice_dns_response_seconds.

    NethVoice proxy (nethesis/ns8-nethvoice-proxy#231)

    Alert Severity Fires when
    ServiceDown critical kamailio, postgres, redis or rtpengine is stopped for 10 minutes (reason=stopped) or restarted at least 3 times in 10 minutes (reason=crash_loop)
    NethVoiceTrunkUnreachable critical for 10 minutes, on a UDP trunk that goes through the proxy, the provider rejects the registration (reason=registration_rejected), does not answer the registration (reason=registration_timeout), or does not answer OPTIONS (reason=qualify_timeout)

    The proxy alerts show the trunk name when NethVoice exports it. Trunks over TCP and TLS are not checked by the proxy yet.

  17. gsanchietti commented on Oct 28, 2026

    @gsanchietti
    gsanchietti
    on Oct 28, 2026
    Member

    Testing versions: nethvoice:1.8.0-testing.55, nethvoice-proxy:1.7.5-testing.11, core:3.24.0, metrics:1.7.0

    Clean install:

    curl https://raw.githubusercontent.com/NethServer/ns8-core/main/core/install.sh > install.sh
bash install.sh ghcr.io/nethserver/core:latest ghcr.io/nethserver/core:3.24.0
add-module ghcr.io/nethserver/metrics:1.7.0 1
add-module ghcr.io/nethesis/nethvoice-proxy:1.7.5-testing.11 1
add-module ghcr.io/nethesis/nethvoice:1.8.0-testing.55 1

    Update existing install:

    update-core ghcr.io/nethserver/core:3.24.0
update-module ghcr.io/nethserver/metrics:1.7.0 metrics1
update-module ghcr.io/nethesis/nethvoice-proxy:1.7.5-testing.11 nethvoice-proxy1
update-module ghcr.io/nethesis/nethvoice:1.8.0-testing.55 nethvoice1

    (core update docs, module update docs)

    Before the alert tests, set up email or portal notifications in the Metrics app so alerts can be received.

    Test case 1: Clean install, alerts off by default

    1. Install with the clean install commands above and complete the NethVoice and NethVoice proxy setup.
    2. Open the NethVoice Settings page: an "Alerts" card is shown and the switch is off.
    3. Open the NethVoice proxy Settings page: same "Alerts" card, switch off.
    4. Stop a NethVoice service (e.g. Tancredi) for more than 15 minutes: no alert is received.
    5. Start the service again.

    Test case 2: Update from previous release

    1. Start from NethVoice 1.7.14 and NethVoice proxy 1.7.4 with working phones and trunks.
    2. Run the update commands above.
    3. Phones, trunks, calls and CTI still work.
    4. The "Alerts" card is shown in Settings of both modules, switch off.
    5. Turn alerts on in both modules: the save completes without errors.

    Test case 3: Turn alerts on and off

    1. Turn alerts on in NethVoice. Stop Tancredi (or another service) for more than 10 minutes: a "service down" alert for Tancredi is received, showing the NethVoice instance and node.
    2. Turn on Tancredi: the alert clears
    3. Turn alerts off again and sttop Tancredi: no alert stays active.

    Test case 4: Unused services are not watched

    1. Pick services not in use on this install (e.g. SAML2/OAuth2 login, or Satellite with transcription off).
    2. With alerts on, wait more than 15 minutes: no "service down" alert is received for them.

    Test case 5: Trunk that skips the proxy is unreachable

    1. In NethVoice, create a trunk that registers to the provider without going through the proxy, with a wrong password.
    2. After more than 10 minutes: a "trunk unreachable" alert with reason "registration_failed" is received, showing the trunk name.
    3. Fix the password: the alert clears.
    4. Create an IP trunk (no registration) that skips the proxy and points to an address that never answers. After more than 10 minutes: an alert with reason "qualify_failed" is received.
    5. Create a trunk that waits for the remote side to register ("receive") and leave the remote side unregistered. After more than 10 minutes: an alert with reason "not_registered" is received.

    Test case 6: UDP trunk through the proxy is unreachable

    1. With alerts on in NethVoice proxy, create a UDP trunk through the proxy that registers to the provider, with a wrong password.
    2. After more than 10 minutes: a proxy alert with reason "registration_rejected" is received, showing the NethVoice trunk name.
    3. Point a registering trunk to an address that never answers: an alert with reason "registration_timeout" is received.
    4. Point a UDP IP trunk to an address that never answers: an alert with reason "qualify_timeout" is received.
    5. Fix each trunk: its alert clears.
    6. Create a TCP or TLS trunk through the proxy and break it: no proxy alert is received (not supported yet).

    Test case 7: Trunk name with a space

    1. In the NethVoice wizard, type a trunk name with a space: a warning is shown under the field.
    2. Save an IP trunk with a space in its name and no From User. Wait more than 10 minutes.
    3. A "trunk not monitored" warning alert is received for that trunk.
    4. Rename the trunk without spaces and apply the configuration: the alert clears.

    Test case 8: DNS name not resolved or pointing to the wrong address

    1. With alerts on, make the node DNS server unable to resolve the NethVoice name (remove the record, or point the node to a DNS server that does not know it).
    2. After more than 15 minutes: a DNS failure alert with reason "not_resolved" is received for the NethVoice name.
    3. Repeat for the CTI name.
    4. Make the proxy name resolve to an address that is not the proxy public or private IP. After more than 15 minutes: a DNS failure alert with reason "ip_mismatch" is received.
    5. Point the node to a DNS server that does not answer at all. After more than 15 minutes: DNS failure alerts with reason "no_answer" are received.
    6. Restore the correct DNS: all DNS alerts clear.

    Test case 9 (optional): Slow DNS
    Needs a DNS server that answers slowly.

    1. Make the node DNS server answer in more than 3 but less than 5 seconds (e.g. add a delay on the network path to it).
    2. After more than 15 minutes: a "DNS slow" warning alert is received.
    3. No DNS failure alert is received for the same name at the same time.
    4. Remove the delay: the alert clears.

    Test case 10: Invalid AI key

    1. Turn on call or voicemail transcription and set a wrong Deepgram key.
    2. After more than 10 minutes: an "AI key invalid" alert is received for Deepgram.
    3. Set a valid key: the alert clears.
    4. Repeat with a wrong OpenAI key.
    5. Turn transcription off, keeping the wrong key: no alert is received.
    6. With transcription on, block the node from reaching the AI provider: no alert is received.

    Test case 11 (optional): AI credit used up
    Needs an AI provider key with no credit left.

    1. With transcription on, use a valid key whose account has no credit left and make a call that is transcribed.
    2. An "AI credit exhausted" alert is received.
    3. Switch to a key with credit and make another transcribed call: the alert clears.

    Test case 12: Backup restore, clone and removal

    1. Turn alerts on in NethVoice and take a backup.
    2. Restore the backup as a new instance: alerts are still on, and a stopped service raises an alert for the restored instance.
    3. Clone NethVoice to another node: alerts are still on and alerts show the new instance and node.
    4. Remove a NethVoice instance: no more alerts from it. Alerts of other NethVoice instances and of NethVoice proxy still work.
  18. gsanchietti
    added
    testingPackages are available from testing repositories
    on Oct 28, 2026
  19. gsanchietti
    removed their assignment
    on Oct 28, 2026
  20. nethbot
    moved this from In Progress to Testing in NethVoiceon Oct 28, 2026
  21. gsanchietti
    self-assigned this
    on Oct 28, 2026
  22. edospadoni
    assigned
    andrea-marchionni
    and unassigned
    gsanchietti
    on Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

nethvoiceBug or features releted to the NethVoice projecttestingPackages are available from testing repositories

Type

Backend

Projects

  • NethVoice
    Status
    Testing

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions