You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Slack notifications for inline alerts fail when the chart-explorer link passes Slack's 3000-character block limit #3452
Slack notifications for inline alerts fail when the alert's chart config is large. The alert does fire, but HyperDX records WEBHOOK_ERROR: Failed to send notification to webhook "<name>" and nothing reaches Slack. The same alert set up as a tile alert posts fine.
Cause
For an inline alert, buildAlertMessageTemplateHdxLink (packages/api/src/tasks/checkAlerts/template.ts) links to the chart explorer with the alert's whole chartConfig URL-encoded in the query string (buildChartExplorerLink). Saved-search and tile alerts link to /search/<id> or the dashboard, which are short.
The Slack transport (packages/api/src/tasks/checkAlerts/transports/slack.ts) puts the link, the title and the body into a single section block:
Slack limits a section block's text to 3000 characters, and the URL counts toward that limit even though Slack only shows the title. So once the encoded config, the title and the body pass 3000 characters, Slack rejects the message.
Example (HyperDX 2.39.1)
An inline builder alert on a metrics source with 8 histogram series (quantile p99, one metricName each):
Each series becomes about 280 characters of the link, because the stored config adds aggCondition, aggConditionLanguage, isDelta and similar fields and URL encoding roughly multiplies JSON by 1.4.
With series aliases, a descriptive name and display settings (numberFormat, fitYAxisToData), the link was about 3100 characters. Every notification failed with WEBHOOK_ERROR.
After removing the aliases, display settings and most of the name, the link was about 2580 characters and the notification posted.
A raw SQL inline alert with about 1700 characters of SQL produces a link of about 2500 characters and is right at the edge.
So a perfectly ordinary alert silently loses its notifications. The only sign is an executionErrors entry on the alert.
Suggestions
Any of these would fix it:
Link inline alerts to something short, for example the alert itself (an alerts page or anchor by alert id), or a server-side stored chart that opens by id, instead of encoding the whole config in the URL.
In the Slack transport, if the section text would pass 3000 characters, fall back to a shorter link (the alerts page) or put the link in a separate block or button. A button url allows up to 3000 characters on its own.
At minimum, warn when saving an inline alert whose notification link would exceed the Slack limit, and say in the WEBHOOK_ERROR that the message was too long.
Activity
fix(api): prevent Slack notification failures when alert message exce…