You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
emerging_signals: compare both windows in one Drain tree, and report new variants of baseline patterns #3556
Follow-up to #3544 (option 2). #3551 now covers option 1 (masking) only.
Problem
emerging_signals mines the baseline and the current window in two separate Drain trees and matches them by template text. Masking (#3551) removes dates and ids, but one false positive stays: a token that is constant in one window and varies in the other. For example, every baseline line says connect failed host=db-1 retry later, while the current window has host=db-1, host=db-2, …. The baseline template keeps host=db-1, the current one becomes host=<*>, so the same message is reported once as emerging and once as disappeared.
Why one shared tree alone isn't enough
Mining both windows in one tree fixes that, but review on #3551 found the cost: Drain merges any same-length line with ≥ 40% token overlap (drainSimTh 0.4) into an existing cluster. With a baseline user login failed: invalid password, a new user login failed: account locked joins the baseline's cluster (user login failed: <*> <*>), so a genuinely new error is no longer reported as new.
Proposal
One tree, mined in a fixed order (baseline first, then current), compared by cluster id.
Per-row cluster assignment from minePatterns, so counts come from Drain itself.
New-variant detection: a current row whose cluster was created by the baseline, but which doesn't match that cluster's baseline template (token compare, no regex), is reported as new_variant, with the baseline pattern and its share.
The baseline window excludes its end, so a row at the boundary is counted once.
Measured on test data with 100 changed patterns: the variant step takes about 10 ms per 10K rows, and the whole call went from 496 ms to 221 ms.
Activity