Skip to content

Native KLL sketch functions (kll_*): audit the crate against Java and implement #7399

Open

Description

@andygrove

What is the problem the feature request solves?

Part of #7397.

Spark 4.1+ provides KLL quantile sketches in bigint, float and double variants: kll_sketch_agg_*, kll_merge_agg_*, kll_sketch_merge_*, kll_sketch_get_quantile_*, kll_sketch_get_rank_*, kll_sketch_get_n_* and kll_sketch_to_string_*. None of them run natively in Comet. The crate's kll feature is new in datasketches 0.6.0.

Describe the potential solution

  1. Score the family with suggest-native-expression first.
  2. Audit the crate's kll feature against DataSketches-Java 6.2.0 using committed fixtures: serialized sketches of each item type, merge results, quantile and rank answers, and the to_string output. KLL compaction is randomized in Java, so work out which results are deterministic enough to compare exactly and which can only be compared within the rank error bound. The crate's KllFloat adapter rejects NaN, so check how Spark treats NaN and infinities.
  3. Implement the serde and native functions once the audit shows the serialized form is read in both directions, opt-in until then.
  4. Add SQL file tests, including a Spark-produced sketch column read by Comet and the reverse.

Additional context

The new crate feature is young (first release in 0.6.0), so read its open issues before relying on it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions