From 6adaa9412569e0ae7a5016d0aeca6895d157967e Mon Sep 17 00:00:00 2001 From: Boris Tyshkevich Date: Thu, 12 Nov 2026 20:20:23 +0100 Subject: [PATCH] FINAL clause speed: refresh for ClickHouse 23.2-26.10, add skip-index and lazy FINAL notes Fix wrong versions (23.3, 24.5/24.6 vertical FINAL, 25.5/25.6 skip indexes, 25.11, row policy defaults), add missing changelog entries, a skip-index exact-mode note (PR 121799) and a lazy FINAL vs lazy materialization section. Benchmarks and methodology are in the PR description. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01D1KN5umrvcVcdzrL77H4k3 --- .../altinity-kb-final-clause-speed.md | 55 +++++++++++++++---- 1 file changed, 45 insertions(+), 10 deletions(-) diff --git a/content/en/altinity-kb-queries-and-syntax/altinity-kb-final-clause-speed.md b/content/en/altinity-kb-queries-and-syntax/altinity-kb-final-clause-speed.md index fc156f2811..c4e2d2a71b 100644 --- a/content/en/altinity-kb-queries-and-syntax/altinity-kb-final-clause-speed.md +++ b/content/en/altinity-kb-queries-and-syntax/altinity-kb-final-clause-speed.md @@ -12,15 +12,26 @@ description: > * Since 20.5 - final can be parallel, see [https://github.com/ClickHouse/ClickHouse/pull/10463](https://github.com/ClickHouse/ClickHouse/pull/10463) * Since 20.10 - you can use `do_not_merge_across_partitions_select_final` setting. See [https://github.com/ClickHouse/ClickHouse/pull/15938](https://github.com/ClickHouse/ClickHouse/pull/15938) and [https://github.com/ClickHouse/ClickHouse/issues/11722](https://github.com/ClickHouse/ClickHouse/issues/11722) * Since 22.6 - final even more parallel, see [https://github.com/ClickHouse/ClickHouse/pull/36396](https://github.com/ClickHouse/ClickHouse/pull/36396) -* Since 22.8 - final doesn't read excessive data, see [https://github.com/ClickHouse/ClickHouse/pull/47801](https://github.com/ClickHouse/ClickHouse/pull/47801) +* Since 23.2 - `WHERE` conditions on the sorting key are moved to `PREWHERE` for FINAL queries, see [https://github.com/ClickHouse/ClickHouse/pull/38950](https://github.com/ClickHouse/ClickHouse/pull/38950); `final=1` setting applies FINAL to every table +* Since 23.3 - final doesn't read excessive data, see [https://github.com/ClickHouse/ClickHouse/pull/47801](https://github.com/ClickHouse/ClickHouse/pull/47801) * Since 23.5 - final use less memory, see [https://github.com/ClickHouse/ClickHouse/pull/50429](https://github.com/ClickHouse/ClickHouse/pull/50429) * Since 23.9 - final doesn't read PK columns if unneeded ie only one part in partition, see [https://github.com/ClickHouse/ClickHouse/pull/53919](https://github.com/ClickHouse/ClickHouse/pull/53919) * Since 23.12 - final applied only for intersecting ranges of parts, see [https://github.com/ClickHouse/ClickHouse/pull/58120](https://github.com/ClickHouse/ClickHouse/pull/58120) +* Since 23.12 - `do_not_merge_across_partitions_select_final` is derived automatically when the partition key uses only primary key columns, see [https://github.com/ClickHouse/ClickHouse/pull/58218](https://github.com/ClickHouse/ClickHouse/pull/58218) * Since 24.1 - final doesn't compare rows from the same part with level > 0, see [https://github.com/ClickHouse/ClickHouse/pull/58142](https://github.com/ClickHouse/ClickHouse/pull/58142) -* Since 24.1 - final use vertical algorithm (more cache friendly), see [https://github.com/ClickHouse/ClickHouse/pull/54366](https://github.com/ClickHouse/ClickHouse/pull/54366) -* Since 25.6 - final supports skip indexes (`use_skip_indexes_if_final=1` by default) -* Since 25.12 - `apply_prewhere_after_final` and `apply_row_policy_after_final` settings for correct PREWHERE/row policy handling with FINAL +* Since 24.1 - final use vertical algorithm (more cache friendly), see [https://github.com/ClickHouse/ClickHouse/pull/54366](https://github.com/ClickHouse/ClickHouse/pull/54366). Disabled in 24.5 because of a bug, re-enabled in 24.6. +* Since 24.2 - `split_parts_ranges_into_intersecting_and_non_intersecting_final` and `split_intersecting_parts_ranges_into_layers_final` settings allow disabling the 23.12 optimization, see [https://github.com/ClickHouse/ClickHouse/pull/59705](https://github.com/ClickHouse/ClickHouse/pull/59705) +* Since 25.1 - with `optimize_on_insert` parts are written at level 1, so the same-part optimizations also apply to fresh parts, see [https://github.com/ClickHouse/ClickHouse/pull/73132](https://github.com/ClickHouse/ClickHouse/pull/73132) +* Since 25.5 - `use_skip_indexes_if_final_exact_mode` added; since 25.6 FINAL uses skip indexes (`use_skip_indexes_if_final=1` by default) +* Since 25.10 - faster FINAL on ReplacingMergeTree with `is_deleted`, see [https://github.com/ClickHouse/ClickHouse/pull/88090](https://github.com/ClickHouse/ClickHouse/pull/88090) +* Since 25.12 - `apply_row_policy_after_final` and `apply_prewhere_after_final` settings (applied after deduplication); row policy default is 1 in 25.12, 0 in 26.1 and 1 from 26.2 * Since 26.2 - `enable_automatic_decision_for_merging_across_partitions_for_final=1` by default (auto-enables cross-partition optimization when safe) +* Since 26.3 - FINAL no longer prunes partitions by `WHERE` (a stale row could win otherwise). Opt-out: `defer_partition_pruning_after_final=0`, useful when one key never spans partitions, see [https://github.com/ClickHouse/ClickHouse/pull/104705](https://github.com/ClickHouse/ClickHouse/pull/104705) +* Since 26.4 - `query_plan_optimize_lazy_final` (**off by default**), see "Lazy FINAL and lazy materialization" below +* Since 26.6 - `FINAL` on the left-most table of a `JOIN` no longer applies to the other joined tables; `analyzer_compatibility_apply_final_to_all_joined_tables=1` (added in 26.4) restores the old behaviour +* Since 26.7 - default lazy materialization also applies to `FINAL` + filter + small `LIMIT` without `ORDER BY` (backported to 26.5.6+ and 26.6.2+ per the PR; checked in 26.5.7 and 26.6.8, absent in 26.4), see [https://github.com/ClickHouse/ClickHouse/pull/110722](https://github.com/ClickHouse/ClickHouse/pull/110722) +* Since 26.9 - ReplacingMergeTree FINAL can read in reverse order (`ORDER BY key DESC LIMIT n`), see [https://github.com/ClickHouse/ClickHouse/pull/115345](https://github.com/ClickHouse/ClickHouse/pull/115345) +* Since 26.10 - `optimize_move_to_prewhere_if_final=1` by default; faster skip-index exact mode with small granules, see [https://github.com/ClickHouse/ClickHouse/pull/121799](https://github.com/ClickHouse/ClickHouse/pull/121799) ### Partitioning @@ -128,7 +139,7 @@ SELECT sum(value) FROM uint64_table FINAL format JSON; ### Vertical FINAL Algorithm (24.1+) -When `enable_vertical_final=1` (default since 24.1), ClickHouse uses a different deduplication strategy: +When `enable_vertical_final=1` (default since 24.1, except 24.5 where it was disabled due to a bug), ClickHouse uses a different deduplication strategy: - Marks duplicate rows as deleted instead of merging them immediately - Filters deleted rows in a later processing step - Reads different columns from different parts in parallel @@ -137,7 +148,7 @@ This improves performance for queries that read only a subset of columns, as non ### PREWHERE and Row Policies with FINAL (25.12+) -By default, PREWHERE and row policies are applied **before** FINAL deduplication. This can cause incorrect results when: +Row policies are applied **after** FINAL by default (`apply_row_policy_after_final=1`; 25.12 and 26.2+, 26.1 had 0). PREWHERE is by default applied **before** FINAL deduplication. This can cause incorrect results when: - PREWHERE references columns that differ across duplicate rows - Row policies should filter based on the "winning" row values after deduplication @@ -147,9 +158,27 @@ Use these settings when needed: Example problem: if you have `ReplacingMergeTree` with a `deleted` column and PREWHERE filters on it, without `apply_prewhere_after_final=1` you may get wrong results because PREWHERE sees rows before FINAL picks the winner. -### FINAL with skip indexes: -- Both `use_skip_indexes_if_final` and `use_skip_indexes_if_final_exact_mode` are enabled by default since 25.6 -- Skip indexes on PRIMARY KEY columns have lower overhead (no extra rescan needed since 26.1), see [https://github.com/ClickHouse/ClickHouse/pull/78350](https://github.com/ClickHouse/ClickHouse/pull/78350) +### FINAL with skip indexes + +- `use_skip_indexes_if_final` and `use_skip_indexes_if_final_exact_mode` are enabled by default since 25.6 (exact mode was added in 25.5). +- Skip indexes on PRIMARY KEY columns have lower overhead (no extra primary-key intersection check since 25.11), see [PR 88368](https://github.com/ClickHouse/ClickHouse/pull/88368). +- Exact mode adds a recovery pass that also reads granules of other parts overlapping the selected ones, so a newer row version is never missed. Before 26.10 its cost grew with the **total** number of granules; since 26.10 it grows with the **selected** granules ([PR 121799](https://github.com/ClickHouse/ClickHouse/pull/121799)). Results and selected granules are identical. In our test (8M rows, 40 unmerged parts, selective index) it gave 4x at `index_granularity=8` (417 → 103 ms), 1.3x at 64 and nothing at the default 8192; the index type does not matter. +- Check that the index pays off: when it matches about 1 row in 200, exact mode was slower than `use_skip_indexes_if_final=0`. Compare both with `EXPLAIN indexes = 1`. + +### Lazy FINAL and lazy materialization + +Two different optimizations are easy to confuse: + +- **Lazy column materialization** (`query_plan_optimize_lazy_materialization`, on by default): read only the filter and FINAL-merge columns first, fetch the rest for the rows that survive `LIMIT`. Since 26.7 it also works for `FINAL` + `WHERE`/`PREWHERE` + `LIMIT` **without** `ORDER BY`. It needs a filter and `LIMIT` up to `query_plan_max_limit_for_lazy_materialization` (10000); look for `JoinLazyColumnsStep` in `EXPLAIN`. On wide tables (3 x `String(200)`, `SELECT * ... WHERE f = 7 LIMIT 10`) it cut the data read from 899 MB to 98 MB and the time from ~106 ms to ~28 ms; no gain with narrow columns, one part, or `LIMIT 100000`. +- **Lazy FINAL** (`query_plan_optimize_lazy_final`, off by default since 26.4) re-reads the table by a primary key set built from the filtered rows. It helps only when the filter prunes marks (a contiguous key range or a unique id, no small `LIMIT`: ~430 → ~25 ms) and is slower when many rows match (+30-45% at 10-50% selectivity without `LIMIT`, up to 5x with `LIMIT 1`). Keep it off otherwise. + +Tested on 26.4, 26.7, 26.8 and 26.10 with identical results in all combinations (including Nullable primary key, `is_deleted` with PREWHERE, text index filter). Caches were warm, so cold-disk gains may be larger. + +### Gotchas + +- **24.5:** `enable_vertical_final` was off by default due to a bug; fixed in 24.6. +- **SummingMergeTree:** `SELECT ... FINAL` keeps rows whose summed columns are all zero, while background merges and `OPTIMIZE ... FINAL` remove them. In 25.8 the row count also depended on the columns selected (`count()` returned 2, `SELECT a` returned 1 for the same data); this is consistent in 26.8+ (changelog entry for 26.10: [PR 115032](https://github.com/ClickHouse/ClickHouse/pull/115032)). +- **Partition pruning (26.3+):** `WHERE` on the partition key is not used to prune partitions under FINAL, see `defer_partition_pruning_after_final` above. ### Settings reference @@ -160,6 +189,12 @@ Example problem: if you have `ReplacingMergeTree` with a `deleted` column and PR | `max_final_threads` | 0 (auto) | 20.5 | Thread limit for FINAL processing | | `enable_vertical_final` | 1 | 24.1 | Read columns in parallel from different parts | | `use_skip_indexes_if_final` | 1 | 25.6 | Allow skip indexes with FINAL | -| `use_skip_indexes_if_final_exact_mode` | 1 | 25.6 | Rescan newer parts to ensure correctness with skip indexes | +| `use_skip_indexes_if_final_exact_mode` | 1 | 25.5 (default 25.6) | Rescan newer parts to ensure correctness with skip indexes | | `apply_prewhere_after_final` | 0 | 25.12 | Apply PREWHERE after deduplication (needed when PREWHERE references non-PK columns) | +| `apply_row_policy_after_final` | 1 | 25.12 (0 in 26.1) | Apply row policies after deduplication | +| `optimize_move_to_prewhere_if_final` | 1 | 23.2 (default 1 since 26.10) | Move sorting-key conditions to PREWHERE for FINAL queries | +| `defer_partition_pruning_after_final` | 1 | 26.3 | Skip partition pruning under FINAL; set 0 only if a key never spans partitions | +| `analyzer_compatibility_apply_final_to_all_joined_tables` | 0 | 26.4 (default 0 since 26.6) | Apply FINAL on the left-most JOIN table to all joined tables | +| `query_plan_optimize_lazy_final` | 0 | 26.4 | Re-read FINAL data by a primary key set from filtered rows; helps for clustered key ranges / unique ids only | +| `query_plan_optimize_lazy_materialization` | 1 | 26.7 for FINAL + LIMIT without ORDER BY | Fetch non-filter columns only for rows that survive `LIMIT` | | `enable_automatic_decision_for_merging_across_partitions_for_final` | 1 | 26.2 | Auto-enable `do_not_merge_across_partitions_select_final` when partition key is in PK |