Skip to content

Core SQL test harness: no runner retries, Raw header without escapes, heredocs in TopLevelWords - #623

Merged
ccoltx merged 2 commits into
masterfrom
polyglot/core-sql-review-fixes
Nov 19, 2026
Merged

ccoltx merged 2 commits into
masterfrom
polyglot/core-sql-review-fixes

Conversation

@polyglotAI-bot

@polyglotAI-bot polyglotAI-bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Three fixes for the core SQL test harness (tests/clickhouse-test-runner/, merged in #621). A review of the same harness in the C# client (ClickHouse/clickhouse-cs#747) found them; the C++ shim has the same three problems. The change is in the harness only: the library (clickhouse/) does not change.

  1. The runner's own retries are off. The pinned tests/clickhouse-test runs a failed test again when its output has one of its MESSAGES_TO_RETRY (for example "No such file or directory", "Connection reset by peer"), and reports a pass on the retry as OK (check_if_need_retry). So a flaky test can enter the allowlist, and no warning shows it. scripts/run-upstream-tests.sh now gives --dont_retry_failures to the plain run and to the RETRY_FAILED loop, so RETRY_FAILED is the only retry. (The runner defines the flag after a parse_known_args() call, so --help does not show it; it is accepted.)
  2. The header of a Raw TSV format has no escapes. For TSVRawWithNames, TSVRawWithNamesAndTypes and their aliases, the server writes the names and types lines without escapes, as it writes the rows (TabSeparatedRowOutputFormat::writeLine). The shim escaped the header always, so a column name with a tab or a backslash gave a\tb where the server gives a real tab. Runner::WriteBlock now uses the Raw flag of the format for the header too.
  3. TopLevelWords skips heredocs. The statement splitter and the FORMAT-clause finder skip $$...$$ and $tag$...$tag$, but TopLevelWords (used to find the inline data of INSERT ... SELECT ... FORMAT) skipped only quotes and comments: in INSERT INTO t SELECT $$x FROM y$$ ..., the FROM in the heredoc was a top-level word. It now skips a heredoc with HeredocEnd, as the splitter does.

Tests

  • Live.HeaderOfFormatMatchesServer (live, shard 1 of CI): column names with a tab, a backslash, a quote, a newline and non-ASCII text, and an Enum type name with quotes and a backslash, in TSVRawWithNames, TabSeparatedRawWithNamesAndTypes and TSVRawWithNamesAndTypes. The expected text is the server's own output of the same query. Contrast: TSVWithNames and TSVWithNamesAndTypes (escaped header). Without the fix, the three Raw queries fail and the two contrast queries pass.
  • SqlText.TopLevelWordsSkipsLiteralsAndComments (unit): words in quotes, a $$ heredoc, a $tag$ heredoc and comments are skipped; a $ that does not start a heredoc ($a$b$) does not skip the text after it. SplitStatements.Cases has a new case for the effect on the splitter: INSERT INTO t SELECT $$ FROM $$ AS x, format y FROM s; is a statement of its own, not format y followed by INSERT data. Without the fix, the two heredoc cases and the splitter case fail.
  • Local, with the server of the pinned version (26.10.4.207): 51/51 unit and live tests with Clang 18 and with GCC 13 (-Werror); clang-format --dry-run --Werror and shellcheck -x clean; the upstream tests that use a Raw format with names or a heredoc (00397_tsv_format_synonym, 01948_heredoc, 01957_heredoc_more, 03580_heredoc_ambiguity, and 3 others) pass with --dont_retry_failures.
  • CI runs the full allowlist with the flag. No test result changes, so allowlist.txt does not change.

Reference: the C# fixes, ClickHouse/clickhouse-cs#747 (commit bb8ef3af).

🤖 Generated with Claude Code


Note

Low Risk
Changes are confined to the test harness and SQL text helpers; no library or production client paths, with regression tests added.

Overview
The clickhouse-test-runner harness gets three behavioral fixes aligned with upstream clickhouse-client and CI expectations.

Upstream test runner retries are off by default: run-upstream-tests.sh passes --dont_retry_failures to tests/clickhouse-test, so message-based automatic retries no longer turn flaky failures into [ OK ]. Optional RETRY_FAILED remains the only retry path; CI is documented as failing the job on the first failure.

Raw TabSeparated name/type headers are written without escapes when the output format is Raw (format.raw), matching the server for TSVRawWithNames and similar formats instead of always using WriteEscapedString on the header.

TopLevelWords now skips heredocs ($$...$$ and $tag$...$tag$) via HeredocEnd, so words inside heredocs are not treated as top-level SQL (fixing statement split / FORMAT detection for heredoc-heavy INSERT queries).

README and new live/unit tests (HeaderOfFormatMatchesServer, TopLevelWordsSkipsLiteralsAndComments, split case with heredoc) cover these behaviors.

Reviewed by Cursor Bugbot for commit f353f05. Bugbot is set up for automated code reviews on this repo. Configure here.

polyglotAI-bot and others added 2 commits November 19, 2026 13:21
- Turn off the own retries of tests/clickhouse-test
  (--dont_retry_failures). Without it, the runner runs a failed test
  again when its output has one of its MESSAGES_TO_RETRY, and reports a
  pass as OK, so a flaky test can enter the allowlist. RETRY_FAILED is
  now the only retry.
- Write the header of the Raw TSV formats (TSVRawWithNames,
  TabSeparatedRawWithNamesAndTypes, ...) without escapes, as the server
  does (TabSeparatedRowOutputFormat::writeLine). Live test against the
  server, with the escaped formats as a contrast.
- Skip heredocs in TopLevelWords, as the statement splitter does, so
  that the words in a heredoc are not top-level words. Unit test.

No test result of the allowlist changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- README: the run script gives the runner --dont_retry_failures, and a
  Raw format writes the header without escapes.
- A splitter test for the heredoc fix of TopLevelWords: a FROM in a
  heredoc does not make `format y` the start of INSERT data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

AI review gate — AI review passed

Clean review from: Cursor Bugbot.

@polyglotAI-bot
polyglotAI-bot marked this pull request as ready for review November 19, 2026 13:58
@polyglotAI-bot
polyglotAI-bot requested review from a team and slabko as code owners November 19, 2026 13:58

@security-ai-gate-reviewer security-ai-gate-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-approved by the AI review gate on f353f05. Details in the approver's comment on this pull request.

@security-ai-gate-reviewer

Copy link
Copy Markdown

AI review gate — auto-approved

f353f057b05e is approved by security-ai-gate-reviewer[bot], an automated identity, not a human. The approval stands on the required AI review: Cursor Bugbot concluded green on this commit and the AI review gate is SUCCESS (its own comment has the verdict), and the change touches no path that requires a human reviewer. It is bound to this commit: a new push dismisses it and the approver re-evaluates.

History (latest 5 heads, newest first)

  • f353f057b05e approved — green: Cursor Bugbot

@ccoltx
ccoltx merged commit f737a42 into master Nov 19, 2026
56 of 57 checks passed
@ccoltx
ccoltx deleted the polyglot/core-sql-review-fixes branch November 19, 2026 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants