Skip to content

Enable stream_aware_field_types by default #27485

Open

Description

@linuspahl

Summary

Flip the default value of the stream_aware_field_types config option from false to true.

Background

stream_aware_field_types is declared in Configuration.java (streamAwareFieldTypes field, @Parameter(value = "stream_aware_field_types")), currently defaulting to false:

If set to "true", Graylog will periodically investigate indices to figure out which fields are used in which streams. It will make field list in Graylog interface show only fields used in selected streams, but can decrease system performance, especially on systems with great number of streams and fields.

Concretely: when true, MappedFieldTypesServiceImpl filters returned field types to only those associated with the requested stream(s), and the index field type pollers do extra batched per-field stream lookups while polling indices to populate that stream association. When false, that lookup is skipped entirely and fields are returned with an empty stream set — the cheaper but stream-unaware path.

The setting was originally added via PR #13363, shipped in 5.0.0 and has defaulted to false since introduction.

Why change the default

This is an optimization — beyond narrowing the field list shown in the UI to only fields present in the selected streams, it also allows narrowing down field types considered for some queries. It was left off by default out of caution — both around performance (the setting's own docs warn of overhead on installs with many streams/fields) and correctness (the original implementation has a known limitation where a field's stream association can go incomplete if the field appears in more streams than search.max_buckets).

Rollout considerations

  • Since the setting is not being removed, this is non-breaking for anyone who has explicitly set stream_aware_field_types themselves — only installs relying on the default are affected.
  • The shipped misc/graylog.conf template sets stream_aware_field_types=false as an active (non-commented) line, not just as a commented example — this must be updated too, or packaged installs will keep getting the old behavior regardless of the Java-side default.
  • Should be called out in the upgrade notes given it changes background indexing behavior (periodic per-field stream lookups during index polling) that can add overhead on installs with a large number of streams/fields, per the setting's own documentation.

Follow-up this enables

#27473 (readable event priority labels in aggregations) originally typed the priority field based on which streams contained the underlying data — accurate, but only worked when stream_aware_field_types was enabled, so that PR shipped with a fallback that instead types based on the streams targeted by the current query. Once stream_aware_field_types defaults to true, we can revisit switching to the stream-based field-typing approach as the primary path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions