Skip to content

Add explicit date mappings for event_created / event_received_time to the default index mapping #27437

Open

Description

@kroepke

Follow-up from #27258.

IndexMapping.fieldProperties (graylog2-server/src/main/java/org/graylog2/indexer/template/IndexMapping.java:111-136) maps timestamp, streams, source, message, full_message and the gl2_* fields, but not the GIM event timestamps. On index sets without a custom mapping, those fields are left to dynamic mapping.

Since #27258, the collector codec writes these as DateTime objects rather than preformatted strings, and since #27428 all three storage backends serialize them identically as ISO-8601. Graylog never sets date_detection anywhere, so OpenSearch's default (true, with strict_date_optional_time among dynamic_date_formats) applies, and IndexMapping7.dynamicStrings() only matches match_mapping_type: "string". Date detection resolves the type before dynamic templates are consulted, so on a new index thesefields now get mapped date automatically.

That works, but it is implicit and fragile.

Why make it explicit

  1. Dynamic detection is first-document-wins per index.
    If the first document in a fresh index carries a non-ISO value for one of these fields — set by a pipeline rule, or by another input writing the same GIM field — the whole index maps it keyword instead, and range queries silently stop working for that index.
  2. It depends on an OpenSearch default we never assert.
    Nothing in our templates pins date_detection, so a future default change or a cluster-level override would silently turn these back into keywords.
  3. It protects against the inverse of the bug Change Collector system logs mapping #27258 fixed.
    A user-set custom Date mapping renders as bare {"type": "date"} with no format (IndexMapping.java:126), which rejects Graylog's yyyy-MM-dd HH:mm:ss.SSS format. Any writer still emitting the legacy format into such a field produces mapper_parsing_exception and drops the whole document.

Proposal

Add event_created and event_received_time as explicit date fields. Worth considering the wider set of GIM date fields at the same time rather than these two in isolation.

Migration note

Index sets that already hold these as keyword will show a field type conflict once they rotate and new indices map them as date. Nothing is rejected and it clears as old indices age out, but it should be mentioned in the changelog.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions