Skip to content

[trino] Add HLL and KLL sketch SQL functions - #5178

Open
rahulbsw wants to merge 2 commits into
apache:mainfrom
rahulbsw:trino-sketch-sql-functions
Open

rahulbsw wants to merge 2 commits into
apache:mainfrom
rahulbsw:trino-sketch-sql-functions

Conversation

@rahulbsw

@rahulbsw rahulbsw commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Part of #3331. Add Trino SQL functions to build, merge, and query serialized Apache DataSketches HLL/KLL doubles sketches stored in Fluss.

Brief change log

  • Register six fluss_ functions in the connector plugin without replacing Trino's native sketch functions.
  • Use mutable single/grouped accumulator states, memory accounting, and explicit intermediate serialization for partial aggregation.
  • Use DataSketches 9.0.0 for Trino's Java 25 runtime. The 6.2.0/Memory 3.0.2 combination used by the Java-8-compatible server rejects Java 25; the plugin therefore uses its own compatible dependency.
  • Retain the dependency license and document precision, null/empty behavior, and read-only connector limits.

Tests

  • CI regression check: FlussSplitManagerTest and SketchFunctionsTest pass together on JDK 25 (41 tests). The branch includes the upstream physical-table-path fix from [Trino] Temporarily isolate test table paths to unblock CI #5196; the Trino Connector CI job also passes.
  • Seven focused tests pass under Trino 483/JDK 25: SQL build/read and merge, null/invalid input, old-format fixtures, grouped state, intermediate serialization, and independent partial state.
  • HLL estimation and compacted KLL fixtures with 20,000 observations generated by DataSketches 6.2.0 are readable by 9.0.0. Separate JVM probes also confirm 6.2.0 can read newly generated 9.0.0 HLL/KLL bytes.
  • Trino plugin verify, Checkstyle, Spotless, RAT, and diff checks pass for the focused suite. The full existing connector suite was not rerun.
  • Inspected the plugin ZIP and JAR: DataSketches 9.0.0 and its preserved license are present; no legacy Memory dependency is packaged.

API and Format

Adds fluss_hll_build_agg, fluss_hll_merge_agg, fluss_hll_estimate, fluss_kll_double_build_agg, fluss_kll_double_merge_agg, and fluss_kll_double_quantile. Final results use portable sketch bytes (HLL lgK 12, KLL k 200); intermediate state has an internal family marker. Readers return DOUBLE. KLL server-side merging requires #5121. Trino connector writes remain unsupported.

Documentation

Adds SQL examples, Java compatibility details, and sketch semantics to the connector README.

Reviewed with sol-loop: 1 rounds with gpt-6.1-sol on high. Sol verdict: approved

Generative AI disclosure: Yes, OpenAI Codex.

Created with OpenAI Codex in the Codex app; pending maintainer review.

@rahulbsw

Copy link
Copy Markdown
Contributor Author

Pulled in main, including #5196's fix for reusing table paths between tests. The full FlussSplitManagerTest class now passes locally (34 tests), along with all seven sketch tests on JDK 25. Pushed the update for CI to confirm.

@rahulbsw
rahulbsw marked this pull request as ready for review November 13, 2026 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant