You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native KLL sketch functions (kll_*): audit the crate against Java and implement #7399
Spark 4.1+ provides KLL quantile sketches in bigint, float and double variants: kll_sketch_agg_*, kll_merge_agg_*, kll_sketch_merge_*, kll_sketch_get_quantile_*, kll_sketch_get_rank_*, kll_sketch_get_n_* and kll_sketch_to_string_*. None of them run natively in Comet. The crate's kll feature is new in datasketches 0.6.0.
Describe the potential solution
Score the family with suggest-native-expression first.
Audit the crate's kll feature against DataSketches-Java 6.2.0 using committed fixtures: serialized sketches of each item type, merge results, quantile and rank answers, and the to_string output. KLL compaction is randomized in Java, so work out which results are deterministic enough to compare exactly and which can only be compared within the rank error bound. The crate's KllFloat adapter rejects NaN, so check how Spark treats NaN and infinities.
Implement the serde and native functions once the audit shows the serialized form is read in both directions, opt-in until then.
Add SQL file tests, including a Spark-produced sketch column read by Comet and the reverse.
Additional context
The new crate feature is young (first release in 0.6.0), so read its open issues before relying on it.
Activity